Oct 30 21:36:58.205696 ip-10-0-133-5 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:36:58.646530 ip-10-0-133-5 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:36:58.646530 ip-10-0-133-5 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:36:58.646530 ip-10-0-133-5 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:36:58.646530 ip-10-0-133-5 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:36:58.653154 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653064 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:36:58.653154 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653151 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:36:58.653154 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653156 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:36:58.653154 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653159 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:36:58.653154 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653162 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653166 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653169 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653171 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653174 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653176 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653179 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653182 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653184 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653187 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653189 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653192 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653194 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653197 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653199 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653202 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653206 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653210 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653213 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:36:58.653355 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653216 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653219 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653222 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653225 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653228 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653231 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653234 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653236 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653244 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653247 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653250 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653252 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653255 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653257 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653260 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653263 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653266 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653269 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653272 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:36:58.653818 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653274 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653277 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653279 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653282 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653284 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653287 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653289 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653292 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653294 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653297 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653299 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653302 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653304 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653307 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653309 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653312 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653314 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653317 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653320 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653322 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:36:58.654292 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653325 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653327 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653330 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653333 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653335 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653338 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653340 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653343 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653346 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653348 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653350 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653353 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653355 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653358 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653360 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653363 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653365 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653368 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653371 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653373 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:36:58.654773 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653376 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653380 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653382 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653384 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653387 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653389 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653392 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653394 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653397 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653399 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653401 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653404 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653406 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653409 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653424 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653427 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653432 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653436 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653439 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:36:58.655248 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653443 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653446 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653449 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653452 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.653454 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654286 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654295 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654299 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654303 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654307 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654310 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654313 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654316 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654319 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654321 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654325 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654328 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654331 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654334 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:36:58.655735 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654337 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654339 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654342 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654344 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654347 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654350 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654352 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654355 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654358 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654361 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654363 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654366 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654369 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654373 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654376 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654378 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654381 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654383 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654386 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654389 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:36:58.656183 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654391 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654394 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654396 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654399 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654402 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654404 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654406 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654409 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654426 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654432 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654436 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654440 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654443 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654445 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654448 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654451 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654454 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654456 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654459 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:36:58.656684 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654461 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654465 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654468 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654470 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654473 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654476 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654478 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654481 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654484 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654487 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654489 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654492 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654495 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654497 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654500 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654502 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654505 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654507 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654510 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:36:58.657120 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654512 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654515 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654517 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654520 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654523 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654526 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654528 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654531 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654533 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654536 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654539 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654542 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654544 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654547 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654550 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654552 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654555 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654557 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654560 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:36:58.657571 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654562 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654565 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654569 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654573 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654576 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654579 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654582 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654584 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654587 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654590 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654592 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654595 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654597 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654600 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.654602 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656029 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656036 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656040 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656044 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656047 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:36:58.658030 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656050 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656053 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656056 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656059 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656062 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656065 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656067 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656070 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656074 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656076 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656079 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656081 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656084 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656087 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656090 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656092 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656095 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656097 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656100 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656102 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:36:58.658521 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656104 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656107 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656109 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656112 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656114 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656116 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656119 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656121 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656124 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656126 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656129 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656132 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656134 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656137 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656139 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656142 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656145 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656147 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656150 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:36:58.659009 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656154 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656158 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656161 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656164 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656166 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656168 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656171 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656175 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656178 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656181 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656184 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656187 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656189 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656192 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656195 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656198 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656201 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656203 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656206 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:36:58.659497 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656208 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656211 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656213 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656216 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656218 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656221 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656224 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656226 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656229 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656232 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656234 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656237 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656240 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656242 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656244 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656247 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656249 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656251 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656254 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656256 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:36:58.660028 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656258 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656261 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656263 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656266 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656268 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656270 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656272 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656275 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656278 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656280 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656282 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656285 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656288 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656290 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656292 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656295 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656300 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656302 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656305 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656308 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:36:58.660513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656310 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656313 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.656315 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656407 2583 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656429 2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656437 2583 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656442 2583 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656447 2583 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656451 2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656456 2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656460 2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656463 2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656466 2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656469 2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656472 2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656476 2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656479 2583 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656481 2583 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656484 2583 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656487 2583 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656490 2583 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656496 2583 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656498 2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656502 2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:36:58.661018 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656505 2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656508 2583 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656512 2583 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656515 2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656520 2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656523 2583 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656526 2583 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656530 2583 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656533 2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656536 2583 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656539 2583 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656544 2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656547 2583 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656550 2583 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656553 2583 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656556 2583 flags.go:64] FLAG: --enable-server="true" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656558 2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656562 2583 flags.go:64] FLAG: --event-burst="100" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656565 2583 flags.go:64] FLAG: --event-qps="50" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656569 2583 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656571 2583 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656574 2583 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656579 2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656582 2583 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656585 2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:36:58.661582 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656588 2583 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656591 2583 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656594 2583 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656597 2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656599 2583 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656602 2583 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656605 2583 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656608 2583 flags.go:64] FLAG: --feature-gates="" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656612 2583 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656615 2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656618 2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656623 2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656626 2583 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656629 2583 flags.go:64] FLAG: --help="false" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656633 2583 flags.go:64] FLAG: --hostname-override="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656636 2583 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656640 2583 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656643 2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656646 2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656649 2583 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656652 2583 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656655 2583 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656658 2583 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:36:58.662238 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656661 2583 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656664 2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656667 2583 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656670 2583 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656673 2583 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656676 2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656679 2583 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656682 2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656685 2583 flags.go:64] FLAG: --lock-file="" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656689 2583 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656692 2583 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656695 2583 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656700 2583 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656703 2583 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656706 2583 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656709 2583 flags.go:64] FLAG: --logging-format="text" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656712 2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656716 2583 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656719 2583 flags.go:64] FLAG: --manifest-url="" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656722 2583 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656727 2583 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656730 2583 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656734 2583 flags.go:64] FLAG: --max-pods="110" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656737 2583 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656740 2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:36:58.662837 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656744 2583 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656747 2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656750 2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656753 2583 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656756 2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656764 2583 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656767 2583 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656770 2583 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656773 2583 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656776 2583 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656779 2583 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656782 2583 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656785 2583 flags.go:64] FLAG: --port="10250" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656788 2583 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656791 2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0600962a47c11e333" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656794 2583 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656797 2583 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656800 2583 flags.go:64] FLAG: --register-node="true" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656803 2583 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656809 2583 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656812 2583 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656815 2583 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656819 2583 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656823 2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656826 2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656829 2583 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:36:58.663467 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656832 2583 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656835 2583 flags.go:64] FLAG: --runonce="false" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656838 2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656841 2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656844 2583 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656847 2583 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656850 2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656854 2583 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656857 2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656861 2583 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656863 2583 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656866 2583 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656869 2583 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656872 2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656875 2583 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656878 2583 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656880 2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656885 2583 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656888 2583 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656891 2583 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656896 2583 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656899 2583 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656902 2583 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656905 2583 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656907 2583 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:36:58.664053 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656910 2583 flags.go:64] FLAG: --v="2" Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656915 2583 flags.go:64] FLAG: --version="false" Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656919 2583 flags.go:64] FLAG: --vmodule="" Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656925 2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.656928 2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657061 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657065 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657068 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657071 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657074 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657076 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657079 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657082 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657085 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657087 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657090 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657093 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657095 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657098 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657100 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:36:58.664658 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657103 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657105 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657108 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657110 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657113 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657117 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657120 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657123 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657126 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657129 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657132 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657134 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657137 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657139 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657142 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657146 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657148 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657151 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657153 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:36:58.665128 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657156 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657158 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657161 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657163 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657166 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657168 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657171 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657173 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657175 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657178 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657181 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657184 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657186 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657189 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657191 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657193 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657196 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657198 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657202 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657204 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:36:58.665614 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657207 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657209 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657212 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657214 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657216 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657219 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657221 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657224 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657228 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657231 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657233 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657236 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657238 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657242 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657245 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657247 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657251 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657254 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657258 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:36:58.666102 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657260 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657263 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657265 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657268 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657271 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657273 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657276 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657279 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657281 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657284 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657286 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657288 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657291 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657293 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657296 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657298 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657301 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657303 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657306 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657308 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:36:58.666592 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657311 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657314 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657317 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657320 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657322 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657325 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657328 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657331 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657333 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657335 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657338 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657340 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657343 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.657368 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657409 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657427 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:36:58.667073 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657431 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657434 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657437 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657439 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657442 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657444 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657447 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657450 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657453 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657455 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657458 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657460 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657463 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657465 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657468 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657470 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657473 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657477 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657481 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:36:58.667474 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657485 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657488 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657492 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657494 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657499 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657502 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657504 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657507 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657509 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657512 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657514 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657517 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657519 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657522 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657525 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657527 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657530 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657532 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657534 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657537 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:36:58.667914 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657540 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657542 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657544 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657547 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657550 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657552 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657554 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657557 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657559 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657562 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657566 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657568 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657571 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657574 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657576 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657579 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657583 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657586 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657588 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657591 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:36:58.668376 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657593 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657597 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657600 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657602 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657604 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657607 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657609 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657612 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657615 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657619 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657622 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657625 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657628 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657631 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657633 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657636 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657638 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657641 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657643 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:36:58.668846 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657645 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657648 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657650 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657655 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657657 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657660 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657662 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657665 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657667 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657671 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657673 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657676 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657678 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657681 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657684 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657687 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657689 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657692 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657694 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657697 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:36:58.669285 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657699 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657702 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657705 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657707 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657710 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.657712 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.664553 2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.664570 2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.664588 2583 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.664593 2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664648 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664653 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664656 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664659 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664662 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664664 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664667 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664670 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664673 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:36:58.669798 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664676 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664678 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664681 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664683 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664686 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664689 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664691 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664694 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664696 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664699 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664701 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664704 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664706 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664708 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664711 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664714 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664716 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664719 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664722 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664724 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:36:58.670265 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664728 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664731 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664733 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664736 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664738 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664740 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664743 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664746 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664748 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664751 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664753 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664756 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664759 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664761 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664764 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664767 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664769 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664772 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664774 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664777 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:36:58.670755 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664779 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664782 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664784 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664786 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664789 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664791 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664794 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664796 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664799 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664801 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664804 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664807 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664810 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664813 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664816 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664818 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664821 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664825 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664829 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664832 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:36:58.671266 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664834 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664837 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664840 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664842 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664845 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664847 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664849 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664852 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664854 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664857 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664859 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664862 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664864 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664867 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664869 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664872 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664875 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664877 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664880 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664882 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:36:58.671728 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664885 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664887 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664890 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664892 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664895 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664898 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664904 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664907 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664909 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664912 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664914 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664917 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664919 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664923 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664926 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664929 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:36:58.672372 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.664932 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.664959 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665090 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665094 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665098 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665101 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665104 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665106 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665109 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665111 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665114 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665116 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665119 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665122 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665125 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665127 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665130 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:36:58.672975 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665132 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665135 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665138 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665141 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665143 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665146 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665149 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665152 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665154 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665157 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665160 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665162 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665164 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665167 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665169 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665172 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665174 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665177 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665179 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:36:58.673632 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665181 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665184 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665186 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665189 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665191 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665193 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665196 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665198 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665201 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665203 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665206 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665208 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665211 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665213 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665216 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665218 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665221 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665223 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665226 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:36:58.674354 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665228 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665231 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665233 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665236 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665238 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665241 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665243 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665246 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665248 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665250 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665253 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665255 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665257 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665261 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665264 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665266 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665269 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665271 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665274 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:36:58.674832 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665276 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665279 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665281 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665284 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665287 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665290 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665292 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665295 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665297 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665299 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665302 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665304 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665307 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665309 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665311 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665315 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665317 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665319 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665322 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665324 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:36:58.675295 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665327 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665330 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665334 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665337 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665340 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665342 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665345 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665348 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665350 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665353 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665355 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665357 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665360 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665362 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.665384 2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665429 2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:36:58.675896 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665434 2583 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665437 2583 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665440 2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665442 2583 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665445 2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665447 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665450 2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665452 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665455 2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665457 2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665460 2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665462 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665465 2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665468 2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665470 2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665473 2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665475 2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665477 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665480 2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665482 2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665485 2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:36:58.676268 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665487 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665490 2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665492 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665495 2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665498 2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665500 2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665502 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665505 2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665511 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665514 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665516 2583 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665519 2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665521 2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665524 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665526 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665528 2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665531 2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665533 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665536 2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:36:58.676751 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665538 2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665540 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665543 2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665545 2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665549 2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665552 2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665554 2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665556 2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665559 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665561 2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665564 2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665566 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665568 2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665571 2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665574 2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665576 2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665578 2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665581 2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665583 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:36:58.677175 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665586 2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665588 2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665591 2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665594 2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665597 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665599 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665603 2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665606 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665609 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665612 2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665615 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665617 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665620 2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665623 2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665625 2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665628 2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665630 2583 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665632 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665635 2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:36:58.677617 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665637 2583 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665640 2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665642 2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665645 2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665647 2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665650 2583 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665652 2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665655 2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665657 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665660 2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665662 2583 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665665 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665667 2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665670 2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665673 2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665678 2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665680 2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665684 2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665686 2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:36:58.678082 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665689 2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665693 2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665696 2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665699 2583 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665703 2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665705 2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665708 2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:36:58.665711 2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.665760 2583 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.667871 2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.668915 2583 server.go:1014] "Starting client certificate rotation" Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.669007 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.669037 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.672592 2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:36:58.678562 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.675038 2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:36:58.687894 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.687878 2583 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:36:58.692240 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.692226 2583 log.go:25] "Validated CRI v1 image API" Oct 30 21:36:58.693102 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.693079 2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:36:58.696271 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.696253 2583 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80151f60-77cf-4281-9b03-0af3145fca58:/dev/nvme0n1p3 adaa7332-d728-43d7-93e6-c04291639958:/dev/nvme0n1p4] Oct 30 21:36:58.696324 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.696271 2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 21:36:58.701619 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.701511 2583 manager.go:217] Machine: {Timestamp:2026-10-30 21:36:58.69988215 +0000 UTC m=+0.367364661 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119513 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23cfecc3c3ef299507dfb7c067a6d7 SystemUUID:ec23cfec-c3c3-ef29-9507-dfb7c067a6d7 BootID:b02f8f5d-675b-48fd-9dd8-fa171e4352b6 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f9:48:58:33 Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f9:48:58:33 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:83:22:70:99:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:36:58.701619 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.701618 2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:36:58.701760 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.701691 2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:36:58.704953 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.704929 2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:36:58.705083 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.704955 2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-5.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:36:58.705130 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.705086 2583 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:36:58.705130 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.705094 2583 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:36:58.705130 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.705119 2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:36:58.705202 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.705131 2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:36:58.706109 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.706099 2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:36:58.709463 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.709451 2583 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:36:58.709577 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.709568 2583 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:36:58.710434 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.710424 2583 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:36:58.710482 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.710440 2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:36:58.710482 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.710461 2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:36:58.710482 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.710473 2583 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:36:58.710656 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.710484 2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:36:58.710834 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.710818 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:36:58.713337 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.713319 2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:36:58.720530 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.720512 2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:36:58.721564 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.721549 2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:36:58.721637 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.721590 2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:36:58.721777 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.721764 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:36:58.721835 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.721783 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:36:58.721835 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.721791 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:36:58.721835 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.721800 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:36:58.721835 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.721808 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:36:58.721835 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.721817 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:36:58.722426 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.722404 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:36:58.722485 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.722432 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:36:58.722485 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.722444 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:36:58.722485 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.722453 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:36:58.722485 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.722469 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:36:58.722485 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.722482 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:36:58.723243 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.723233 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:36:58.723290 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.723246 2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:36:58.727350 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.727335 2583 server.go:1263] "Started kubelet" Oct 30 21:36:58.727492 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.727439 2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:36:58.727599 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.727437 2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:36:58.727599 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.727537 2583 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:36:58.728152 ip-10-0-133-5 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:36:58.728880 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.728865 2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:36:58.732646 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.732623 2583 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:36:58.733249 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:58.733228 2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 21:36:58.735203 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.735168 2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:36:58.735754 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.735740 2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:36:58.735880 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:58.735854 2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" Oct 30 21:36:58.736400 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736385 2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:36:58.736400 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736381 2583 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:36:58.736552 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736409 2583 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:36:58.736600 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736576 2583 factory.go:55] Registering systemd factory Oct 30 21:36:58.736600 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736592 2583 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:36:58.736691 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736613 2583 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:36:58.736691 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736623 2583 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:36:58.736850 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736835 2583 factory.go:153] Registering CRI-O factory Oct 30 21:36:58.736887 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736853 2583 factory.go:223] Registration of the crio container factory successfully Oct 30 21:36:58.736935 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736907 2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:36:58.736966 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736943 2583 factory.go:103] Registering Raw factory Oct 30 21:36:58.737029 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.736978 2583 manager.go:1199] Started watching for new ooms in manager Oct 30 21:36:58.738068 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.738054 2583 manager.go:319] Starting recovery of all containers Oct 30 21:36:58.751834 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.751815 2583 manager.go:324] Recovery completed Oct 30 21:36:58.755782 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.755767 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:36:58.757583 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.757559 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:36:58.757653 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.757600 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:36:58.757653 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.757612 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:36:58.758203 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.758187 2583 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:36:58.758203 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.758195 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:36:58.758203 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.758200 2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:36:58.758349 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.758220 2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:36:58.758972 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.758957 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:36:58.759030 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.758987 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:36:58.759030 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.758998 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:36:58.759734 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.759722 2583 policy_none.go:50] "Start" Oct 30 21:36:58.759772 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.759739 2583 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:36:58.759772 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.759754 2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:36:58.760996 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.760985 2583 policy_none.go:44] "Start" Oct 30 21:36:58.785007 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.784976 2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:36:58.788198 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.788184 2583 manager.go:341] "Starting Device Plugin manager" Oct 30 21:36:58.789046 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:58.789030 2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:36:58.789090 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.789048 2583 server.go:87] "Starting device plugin registration server" Oct 30 21:36:58.789307 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.789296 2583 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:36:58.789336 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.789314 2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:36:58.789452 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.789400 2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:36:58.789533 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.789510 2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:36:58.789533 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.789518 2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:36:58.790093 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:58.790076 2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:36:58.790168 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:58.790123 2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" Oct 30 21:36:58.877736 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.877702 2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-133-5.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 21:36:58.883585 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.883559 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:36:58.884714 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.884695 2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:36:58.884714 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.884714 2583 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:36:58.884845 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.884732 2583 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:36:58.884845 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:58.884765 2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 21:36:58.890257 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.890241 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:36:58.891017 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.890999 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:36:58.891079 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.891038 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:36:58.891079 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.891049 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:36:58.891079 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.891077 2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:58.928061 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:58.926893 2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-133-5.us-west-2.compute.internal.18e36c5beec51860 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-5.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-133-5.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:36:58.72730736 +0000 UTC m=+0.394789871,LastTimestamp:2026-10-30 21:36:58.72730736 +0000 UTC m=+0.394789871,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-5.us-west-2.compute.internal,}" Oct 30 21:36:58.962898 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.962873 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:36:58.963002 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.962921 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-qrb8c" Oct 30 21:36:58.985313 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.985278 2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal"] Oct 30 21:36:58.985402 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.985346 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:36:58.986311 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.986294 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:36:58.986380 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.986333 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:36:58.986380 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.986346 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:36:58.987933 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.987920 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:36:58.988062 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.988046 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:58.988099 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.988078 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:36:58.988730 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.988706 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:36:58.988730 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.988731 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:36:58.988829 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.988737 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:36:58.988829 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.988745 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:36:58.988829 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.988776 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:36:58.988829 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.988789 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:36:58.990125 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.990107 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:58.990197 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.990137 2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:36:58.990842 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.990821 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:36:58.990842 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.990841 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:36:58.991039 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:58.990856 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:36:59.021598 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:59.021580 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.025743 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:59.025727 2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.029181 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.029150 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:36:59.030872 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.030858 2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:36:59.038986 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.038966 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.039076 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.038993 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ab1a8fb7b394ed5c2ad3173d5294ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"3ab1a8fb7b394ed5c2ad3173d5294ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.039076 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.039008 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.123393 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.123354 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:36:59.124182 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.124164 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:36:59.124901 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.124884 2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.136293 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.136270 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.139962 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.139946 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ab1a8fb7b394ed5c2ad3173d5294ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"3ab1a8fb7b394ed5c2ad3173d5294ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.140040 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.139969 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.140040 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.139988 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.140040 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.140033 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.140159 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.140056 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.140159 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.140059 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ab1a8fb7b394ed5c2ad3173d5294ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"3ab1a8fb7b394ed5c2ad3173d5294ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.207281 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.207223 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:36:59.208075 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.208063 2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.272969 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.272946 2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:36:59.325602 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.325575 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.330902 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.329697 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:36:59.668340 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.668258 2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 21:36:59.668926 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.668466 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:36:59.668926 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.668471 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:36:59.668926 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.668487 2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 21:36:59.668926 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:36:59.668451 2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ac60eab6f532d4004a67394301a17821-2cf05b1ccc71f3c2.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-133-5.us-west-2.compute.internal.18e36c5bf0939073\": read tcp 10.0.133.5:37308->3.151.38.176:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-133-5.us-west-2.compute.internal.18e36c5bf0939073 default 5246 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-133-5.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-133-5.us-west-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-133-5.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:36:58 +0000 UTC,LastTimestamp:2026-10-30 21:36:58.891053506 +0000 UTC m=+0.558536018,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-133-5.us-west-2.compute.internal,}" Oct 30 21:36:59.711052 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.711023 2583 apiserver.go:52] "Watching apiserver" Oct 30 21:36:59.735536 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.735511 2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 21:36:59.967887 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.967793 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:31:59 +0000 UTC" deadline="2028-03-29 02:23:33.323355946 +0000 UTC" Oct 30 21:36:59.967887 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:36:59.967835 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12364h46m33.355526158s" Oct 30 21:37:00.010571 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.010539 2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:37:00.010904 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.010883 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-6knhd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4","openshift-cluster-node-tuning-operator/tuned-hp8q8","openshift-multus/network-metrics-daemon-sbp5h","kube-system/global-pull-secret-syncer-9rfmn","kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal","openshift-image-registry/node-ca-ftmsf","openshift-multus/multus-additional-cni-plugins-hw57b","openshift-network-operator/iptables-alerter-fb8rx","openshift-multus/multus-j6hd8","openshift-ovn-kubernetes/ovnkube-node-2snbw","openshift-dns/node-resolver-ps2p9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-vstcw"] Oct 30 21:37:00.012975 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.012950 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.014384 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.014365 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:00.014527 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.014446 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:00.015946 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.015928 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.017506 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.017472 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.018807 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.018794 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:00.018871 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.018846 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:00.020166 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.020152 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.022216 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.020213 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:00.023881 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.023859 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:00.025384 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.025366 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.026793 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.026778 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.028300 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.028279 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.031518 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.031493 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.033009 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.032993 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.037302 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.037286 2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:37:00.044836 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.044819 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-bin\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.044895 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.044844 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-system-cni-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.044895 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.044862 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mtw\" (UniqueName: \"kubernetes.io/projected/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-kube-api-access-96mtw\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.044895 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.044878 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa67807-4df5-4e6f-a4f9-a378593abf94-tmp-dir\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.045006 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.044913 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkqn\" (UniqueName: \"kubernetes.io/projected/eab1dc97-e877-4b0f-ab0c-406eb137b296-kube-api-access-nxkqn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.045006 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.044961 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-iptables-alerter-script\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.045006 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.044979 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-k8s-cni-cncf-io\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.045142 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045015 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-multus-certs\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.045142 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045044 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-modprobe-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.045142 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045060 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-run\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.045142 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045082 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-socket-dir-parent\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.045142 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045097 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a49810a-da35-4032-b7ac-a6e3c7453d21-host\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.045142 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045112 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6jqv\" (UniqueName: \"kubernetes.io/projected/9a49810a-da35-4032-b7ac-a6e3c7453d21-kube-api-access-q6jqv\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.045372 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045157 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w78lc\" (UniqueName: \"kubernetes.io/projected/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-kube-api-access-w78lc\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.045372 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045195 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysconfig\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.045372 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045239 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-netns\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.045372 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045264 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.045372 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045283 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:00.045372 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045319 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.045372 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045335 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cni-binary-copy\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.045372 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045357 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-host\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045373 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-systemd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045397 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovn-node-metrics-cert\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045412 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-daemon-config\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045427 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nstvb\" (UniqueName: \"kubernetes.io/projected/d757110b-4b02-4857-853b-949b53441428-kube-api-access-nstvb\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045489 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045533 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045562 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-system-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045590 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-multus\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045615 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-kubelet\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-registration-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045722 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045754 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-node-log\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045781 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.045808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045806 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045845 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-conf\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045869 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-sys\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045892 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-var-lib-kubelet\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045913 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-socket-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045943 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045964 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045981 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-etc-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.045999 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046020 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-log-socket\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046035 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-script-lib\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046052 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046079 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkg86\" (UniqueName: \"kubernetes.io/projected/10088b05-b328-4162-8ef6-39e60f240fb3-kube-api-access-wkg86\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046098 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cnibin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046114 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-host-slash\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046131 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/788ef145-30de-43f5-8249-799dbb536b1b-agent-certs\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:00.046398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046155 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/788ef145-30de-43f5-8249-799dbb536b1b-konnectivity-ca\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046176 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046191 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-bin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046231 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa67807-4df5-4e6f-a4f9-a378593abf94-hosts-file\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046252 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tqq2\" (UniqueName: \"kubernetes.io/projected/5aa67807-4df5-4e6f-a4f9-a378593abf94-kube-api-access-2tqq2\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046278 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88l6l\" (UniqueName: \"kubernetes.io/projected/556b7537-074f-46ce-8db5-e1a0a7b94dca-kube-api-access-88l6l\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046298 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-netd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046327 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-config\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046345 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-env-overrides\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046368 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-kubelet-config\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046383 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-etc-kubernetes\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-sys-fs\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046478 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-systemd-units\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046521 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmt7z\" (UniqueName: \"kubernetes.io/projected/9b80443a-852a-4e9e-8a6d-26de01cd2912-kube-api-access-kmt7z\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046552 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-dbus\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046586 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a49810a-da35-4032-b7ac-a6e3c7453d21-serviceca\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046615 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-netns\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.047124 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046648 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-hostroot\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046684 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-kubelet\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046711 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-slash\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046745 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-var-lib-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046784 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-cnibin\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046812 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-kubernetes\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046837 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-systemd\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046868 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-lib-modules\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046911 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-etc-tuned\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046953 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046976 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-conf-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.046992 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-device-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.047052 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-ovn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.047628 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.047066 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-tmp\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.070711 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.070691 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:37:00.070711 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.070691 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:37:00.070884 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.070827 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zlst6\"" Oct 30 21:37:00.070884 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.070839 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:37:00.073198 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.073180 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:37:00.073539 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.073518 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b6pf6\"" Oct 30 21:37:00.073665 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.073605 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:37:00.073758 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.073737 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:37:00.073758 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.073749 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:37:00.073924 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.073778 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:37:00.073924 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.073847 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:37:00.074808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.074792 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hkmjr\"" Oct 30 21:37:00.075105 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.075041 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:37:00.075105 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.075073 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:37:00.075105 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.075102 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:37:00.075310 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.075128 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:37:00.075310 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.075128 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:37:00.084100 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.084086 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:37:00.084175 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.084145 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:37:00.084217 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.084189 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:37:00.084250 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.084227 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lt5wb\"" Oct 30 21:37:00.084370 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.084358 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:37:00.084436 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.084425 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:37:00.147775 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147751 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w78lc\" (UniqueName: \"kubernetes.io/projected/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-kube-api-access-w78lc\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.147870 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147781 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysconfig\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.147870 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147807 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-netns\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.147870 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147851 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-netns\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147878 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147880 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysconfig\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147907 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147929 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147933 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.147972 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cni-binary-copy\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.147999 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.148011 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:00.148037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148000 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-host\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148037 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-host\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.148083 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret podName:843e4020-8588-4ead-adda-ead6c9e0882b nodeName:}" failed. No retries permitted until 2026-10-30 21:37:00.648062439 +0000 UTC m=+2.315544953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret") pod "global-pull-secret-syncer-9rfmn" (UID: "843e4020-8588-4ead-adda-ead6c9e0882b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-systemd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.148131 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs podName:9b80443a-852a-4e9e-8a6d-26de01cd2912 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:00.64811384 +0000 UTC m=+2.315596358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs") pod "network-metrics-daemon-sbp5h" (UID: "9b80443a-852a-4e9e-8a6d-26de01cd2912") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148137 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-systemd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148164 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovn-node-metrics-cert\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148200 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-daemon-config\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148229 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstvb\" (UniqueName: \"kubernetes.io/projected/d757110b-4b02-4857-853b-949b53441428-kube-api-access-nstvb\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148254 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148279 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148302 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-system-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148325 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-multus\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148333 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148351 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-kubelet\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148456 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-system-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.148481 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148471 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-registration-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148477 2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148500 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148525 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-node-log\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148522 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148550 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148551 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cni-binary-copy\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148577 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148585 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-kubelet\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148603 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-conf\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148625 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-multus\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148631 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-sys\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148670 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-var-lib-kubelet\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148685 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-sys\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148701 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-socket-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148734 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148747 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.149266 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148766 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148797 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-etc-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148803 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-node-log\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148821 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148831 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-registration-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148837 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148864 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-var-lib-kubelet\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148873 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148873 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-etc-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148889 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-log-socket\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148911 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-script-lib\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148932 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148954 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkg86\" (UniqueName: \"kubernetes.io/projected/10088b05-b328-4162-8ef6-39e60f240fb3-kube-api-access-wkg86\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148959 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-conf\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148975 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cnibin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148984 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-socket-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.148995 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-host-slash\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.150117 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149027 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-host-slash\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149029 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/788ef145-30de-43f5-8249-799dbb536b1b-agent-certs\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149421 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149508 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-script-lib\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149532 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149556 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-log-socket\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149556 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149599 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/788ef145-30de-43f5-8249-799dbb536b1b-konnectivity-ca\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149604 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cnibin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149626 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149652 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-bin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149687 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa67807-4df5-4e6f-a4f9-a378593abf94-hosts-file\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqq2\" (UniqueName: \"kubernetes.io/projected/5aa67807-4df5-4e6f-a4f9-a378593abf94-kube-api-access-2tqq2\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149755 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l6l\" (UniqueName: \"kubernetes.io/projected/556b7537-074f-46ce-8db5-e1a0a7b94dca-kube-api-access-88l6l\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149793 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-netd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149818 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-config\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149832 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-bin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.150978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149832 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149856 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-env-overrides\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149884 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-kubelet-config\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149906 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-netd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149930 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-etc-kubernetes\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149967 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-sys-fs\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149996 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-systemd-units\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150023 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmt7z\" (UniqueName: \"kubernetes.io/projected/9b80443a-852a-4e9e-8a6d-26de01cd2912-kube-api-access-kmt7z\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150055 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-dbus\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150067 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-kubelet-config\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.149817 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa67807-4df5-4e6f-a4f9-a378593abf94-hosts-file\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150083 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a49810a-da35-4032-b7ac-a6e3c7453d21-serviceca\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150120 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-netns\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150139 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/788ef145-30de-43f5-8249-799dbb536b1b-konnectivity-ca\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150150 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-hostroot\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150179 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-kubelet\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150205 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-slash\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150231 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-var-lib-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.151863 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150256 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-cnibin\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150280 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-kubernetes\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150296 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-env-overrides\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150306 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-systemd\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150024 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-etc-kubernetes\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150358 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-slash\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150390 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-netns\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150395 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-sys-fs\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150422 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-hostroot\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150431 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-dbus\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150447 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-systemd-units\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150470 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-kubelet\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150494 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-cnibin\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150517 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-var-lib-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150520 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a49810a-da35-4032-b7ac-a6e3c7453d21-serviceca\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150541 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-kubernetes\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150543 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-systemd\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150578 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-lib-modules\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.152715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150608 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-etc-tuned\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150636 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150661 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-conf-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150687 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-lib-modules\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150685 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-device-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150730 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-ovn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150738 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-conf-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150770 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-tmp\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150788 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-device-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150796 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-bin\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150825 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-system-cni-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150831 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-ovn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150851 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mtw\" (UniqueName: \"kubernetes.io/projected/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-kube-api-access-96mtw\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150879 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa67807-4df5-4e6f-a4f9-a378593abf94-tmp-dir\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150905 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkqn\" (UniqueName: \"kubernetes.io/projected/eab1dc97-e877-4b0f-ab0c-406eb137b296-kube-api-access-nxkqn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150931 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-iptables-alerter-script\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-k8s-cni-cncf-io\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.153618 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151014 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-multus-certs\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.150879 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-bin\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151040 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-modprobe-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151064 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-run\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151082 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-system-cni-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151087 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-socket-dir-parent\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151128 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a49810a-da35-4032-b7ac-a6e3c7453d21-host\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151137 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-socket-dir-parent\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151150 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151157 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jqv\" (UniqueName: \"kubernetes.io/projected/9a49810a-da35-4032-b7ac-a6e3c7453d21-kube-api-access-q6jqv\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151200 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-k8s-cni-cncf-io\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151015 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-config\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151384 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-multus-certs\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151421 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a49810a-da35-4032-b7ac-a6e3c7453d21-host\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151532 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-modprobe-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151588 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-run\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151731 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-iptables-alerter-script\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.151805 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa67807-4df5-4e6f-a4f9-a378593abf94-tmp-dir\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.154525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.152505 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovn-node-metrics-cert\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.155431 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.152656 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/788ef145-30de-43f5-8249-799dbb536b1b-agent-certs\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:00.155431 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.153940 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-etc-tuned\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.155431 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.153981 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-tmp\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.176899 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.176873 2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 21:37:00.177027 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.176917 2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zvnsc" Oct 30 21:37:00.179866 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.179845 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:37:00.179941 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.179922 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:37:00.180002 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.179938 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:37:00.180070 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180003 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:37:00.180070 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180002 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:37:00.180301 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180281 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:37:00.180301 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180278 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:37:00.180301 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180305 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j5bph\"" Oct 30 21:37:00.180531 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180339 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t84mq\"" Oct 30 21:37:00.180531 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180341 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:37:00.180627 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180619 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5bth\"" Oct 30 21:37:00.180684 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180617 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hcvt7\"" Oct 30 21:37:00.180684 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.180674 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gfsxs\"" Oct 30 21:37:00.189318 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.189298 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-daemon-config\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.212606 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.212583 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:37:00.212606 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.212607 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:37:00.212757 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.212616 2583 projected.go:196] Error preparing data for projected volume kube-api-access-62lrs for pod openshift-network-diagnostics/network-check-target-vstcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:00.212757 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.212677 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs podName:708926c2-26a2-4888-ae37-a4c157a07e94 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:00.712660484 +0000 UTC m=+2.380143011 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-62lrs" (UniqueName: "kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs") pod "network-check-target-vstcw" (UID: "708926c2-26a2-4888-ae37-a4c157a07e94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:00.215164 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.215133 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstvb\" (UniqueName: \"kubernetes.io/projected/d757110b-4b02-4857-853b-949b53441428-kube-api-access-nstvb\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.215258 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.215245 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w78lc\" (UniqueName: \"kubernetes.io/projected/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-kube-api-access-w78lc\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.216005 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.215987 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkg86\" (UniqueName: \"kubernetes.io/projected/10088b05-b328-4162-8ef6-39e60f240fb3-kube-api-access-wkg86\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.216630 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.216607 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmt7z\" (UniqueName: \"kubernetes.io/projected/9b80443a-852a-4e9e-8a6d-26de01cd2912-kube-api-access-kmt7z\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:00.216737 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.216724 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l6l\" (UniqueName: \"kubernetes.io/projected/556b7537-074f-46ce-8db5-e1a0a7b94dca-kube-api-access-88l6l\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.216873 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.216854 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqq2\" (UniqueName: \"kubernetes.io/projected/5aa67807-4df5-4e6f-a4f9-a378593abf94-kube-api-access-2tqq2\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.222361 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.222304 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mtw\" (UniqueName: \"kubernetes.io/projected/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-kube-api-access-96mtw\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.222464 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.222449 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkqn\" (UniqueName: \"kubernetes.io/projected/eab1dc97-e877-4b0f-ab0c-406eb137b296-kube-api-access-nxkqn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.222665 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.222642 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jqv\" (UniqueName: \"kubernetes.io/projected/9a49810a-da35-4032-b7ac-a6e3c7453d21-kube-api-access-q6jqv\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.358907 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.358871 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:37:00.387181 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.387153 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:00.405449 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.405411 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:37:00.413069 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.413047 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:00.415324 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.415298 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:37:00.425520 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.425495 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-j6hd8" Oct 30 21:37:00.443760 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.443733 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:37:00.448265 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.448247 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:37:00.468233 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.468206 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:37:00.539500 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.539390 2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:37:00.561645 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:00.561606 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8f9506264df06e876fee4a0052af02.slice/crio-35ec724587d333a9d8f43a27f5c4330b96293de7b2a9f7870e5f6a90dfaa5a17 WatchSource:0}: Error finding container 35ec724587d333a9d8f43a27f5c4330b96293de7b2a9f7870e5f6a90dfaa5a17: Status 404 returned error can't find the container with id 35ec724587d333a9d8f43a27f5c4330b96293de7b2a9f7870e5f6a90dfaa5a17 Oct 30 21:37:00.562064 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:00.562043 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab1a8fb7b394ed5c2ad3173d5294ab9.slice/crio-3843cedccf2e1add68c02348e56020745d39f4044f652b4cf3350c4ea3ab6c5d WatchSource:0}: Error finding container 3843cedccf2e1add68c02348e56020745d39f4044f652b4cf3350c4ea3ab6c5d: Status 404 returned error can't find the container with id 3843cedccf2e1add68c02348e56020745d39f4044f652b4cf3350c4ea3ab6c5d Oct 30 21:37:00.569514 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.569492 2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:37:00.655148 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.655114 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:00.655148 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.655148 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:00.655373 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.655267 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:00.655373 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.655274 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:00.655373 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.655319 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret podName:843e4020-8588-4ead-adda-ead6c9e0882b nodeName:}" failed. No retries permitted until 2026-10-30 21:37:01.655303699 +0000 UTC m=+3.322786204 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret") pod "global-pull-secret-syncer-9rfmn" (UID: "843e4020-8588-4ead-adda-ead6c9e0882b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:00.655373 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.655335 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs podName:9b80443a-852a-4e9e-8a6d-26de01cd2912 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:01.655327977 +0000 UTC m=+3.322810477 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs") pod "network-metrics-daemon-sbp5h" (UID: "9b80443a-852a-4e9e-8a6d-26de01cd2912") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:00.756375 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.756331 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:00.757030 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.756512 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:37:00.757030 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.756534 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:37:00.757030 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.756545 2583 projected.go:196] Error preparing data for projected volume kube-api-access-62lrs for pod openshift-network-diagnostics/network-check-target-vstcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:00.757030 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:00.756600 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs podName:708926c2-26a2-4888-ae37-a4c157a07e94 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:01.75658532 +0000 UTC m=+3.424067821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-62lrs" (UniqueName: "kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs") pod "network-check-target-vstcw" (UID: "708926c2-26a2-4888-ae37-a4c157a07e94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:00.889160 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.889065 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"3ab1a8fb7b394ed5c2ad3173d5294ab9","Type":"ContainerStarted","Data":"3843cedccf2e1add68c02348e56020745d39f4044f652b4cf3350c4ea3ab6c5d"} Oct 30 21:37:00.890149 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.890123 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"1d8f9506264df06e876fee4a0052af02","Type":"ContainerStarted","Data":"35ec724587d333a9d8f43a27f5c4330b96293de7b2a9f7870e5f6a90dfaa5a17"} Oct 30 21:37:00.904800 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.904778 2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:37:00.924521 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:00.924488 2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:37:01.179781 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.179680 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:32:00 +0000 UTC" deadline="2028-08-15 06:13:21.18711424 +0000 UTC" Oct 30 21:37:01.179781 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.179723 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15704h36m20.007393969s" Oct 30 21:37:01.294801 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.294771 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788ef145_30de_43f5_8249_799dbb536b1b.slice/crio-abf664d805b9cc7669cee12830734f25b3feab54e406903a96c5461903ef7c4f WatchSource:0}: Error finding container abf664d805b9cc7669cee12830734f25b3feab54e406903a96c5461903ef7c4f: Status 404 returned error can't find the container with id abf664d805b9cc7669cee12830734f25b3feab54e406903a96c5461903ef7c4f Oct 30 21:37:01.322668 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.322640 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a49810a_da35_4032_b7ac_a6e3c7453d21.slice/crio-49ab811d4b3a6d0e71909de5e5344f69b2b09fee4400496ef0d5f4fec1e3bf8b WatchSource:0}: Error finding container 49ab811d4b3a6d0e71909de5e5344f69b2b09fee4400496ef0d5f4fec1e3bf8b: Status 404 returned error can't find the container with id 49ab811d4b3a6d0e71909de5e5344f69b2b09fee4400496ef0d5f4fec1e3bf8b Oct 30 21:37:01.333466 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.333441 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddc20de6_3c22_4dc4_8b56_9bfd7e9ac9dc.slice/crio-039181eb63ea760ad52c55d4bf65f406ccd3f46ba3ed7e7183d0622396ab338c WatchSource:0}: Error finding container 039181eb63ea760ad52c55d4bf65f406ccd3f46ba3ed7e7183d0622396ab338c: Status 404 returned error can't find the container with id 039181eb63ea760ad52c55d4bf65f406ccd3f46ba3ed7e7183d0622396ab338c Oct 30 21:37:01.361763 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.361731 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab1dc97_e877_4b0f_ab0c_406eb137b296.slice/crio-b75ca48e5a4ea8bdca39ae0aed91953075b66d381845b9ff8b2236b757e79364 WatchSource:0}: Error finding container b75ca48e5a4ea8bdca39ae0aed91953075b66d381845b9ff8b2236b757e79364: Status 404 returned error can't find the container with id b75ca48e5a4ea8bdca39ae0aed91953075b66d381845b9ff8b2236b757e79364 Oct 30 21:37:01.395130 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.395099 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa67807_4df5_4e6f_a4f9_a378593abf94.slice/crio-bd3fed9922a28fad2449285780c43ced8f2188394a6fabe58ea5fab5f6777a05 WatchSource:0}: Error finding container bd3fed9922a28fad2449285780c43ced8f2188394a6fabe58ea5fab5f6777a05: Status 404 returned error can't find the container with id bd3fed9922a28fad2449285780c43ced8f2188394a6fabe58ea5fab5f6777a05 Oct 30 21:37:01.420838 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.420797 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd757110b_4b02_4857_853b_949b53441428.slice/crio-d7c9fe47ec8e23118c1da19d2c76d1d5178a1792da5a3c965b8bff3f51b5dc76 WatchSource:0}: Error finding container d7c9fe47ec8e23118c1da19d2c76d1d5178a1792da5a3c965b8bff3f51b5dc76: Status 404 returned error can't find the container with id d7c9fe47ec8e23118c1da19d2c76d1d5178a1792da5a3c965b8bff3f51b5dc76 Oct 30 21:37:01.476513 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.476466 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10088b05_b328_4162_8ef6_39e60f240fb3.slice/crio-52d00644bd4350dd3c1955975d721d6e21274bf684f44f3acc5e614bf27e4d5a WatchSource:0}: Error finding container 52d00644bd4350dd3c1955975d721d6e21274bf684f44f3acc5e614bf27e4d5a: Status 404 returned error can't find the container with id 52d00644bd4350dd3c1955975d721d6e21274bf684f44f3acc5e614bf27e4d5a Oct 30 21:37:01.477156 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.477129 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod556b7537_074f_46ce_8db5_e1a0a7b94dca.slice/crio-cf97dafb596cc66eff58442e40d8f56d0468e9994a14f72fb0c3a30b1231b4a0 WatchSource:0}: Error finding container cf97dafb596cc66eff58442e40d8f56d0468e9994a14f72fb0c3a30b1231b4a0: Status 404 returned error can't find the container with id cf97dafb596cc66eff58442e40d8f56d0468e9994a14f72fb0c3a30b1231b4a0 Oct 30 21:37:01.537741 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:01.537712 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d7df9d_6dbd_4b9f_a679_00d8d07db782.slice/crio-5fb865af1615becd8b256e62b2c5e48e5844497a001664efd1fb5888170ddaeb WatchSource:0}: Error finding container 5fb865af1615becd8b256e62b2c5e48e5844497a001664efd1fb5888170ddaeb: Status 404 returned error can't find the container with id 5fb865af1615becd8b256e62b2c5e48e5844497a001664efd1fb5888170ddaeb Oct 30 21:37:01.664953 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.664916 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:01.664953 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.664956 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:01.665162 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.665090 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:01.665162 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.665159 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs podName:9b80443a-852a-4e9e-8a6d-26de01cd2912 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:03.665144459 +0000 UTC m=+5.332626964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs") pod "network-metrics-daemon-sbp5h" (UID: "9b80443a-852a-4e9e-8a6d-26de01cd2912") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:01.665251 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.665090 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:01.665251 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.665212 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret podName:843e4020-8588-4ead-adda-ead6c9e0882b nodeName:}" failed. No retries permitted until 2026-10-30 21:37:03.665200495 +0000 UTC m=+5.332682998 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret") pod "global-pull-secret-syncer-9rfmn" (UID: "843e4020-8588-4ead-adda-ead6c9e0882b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:01.766226 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.766108 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:01.766817 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.766282 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:37:01.766817 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.766306 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:37:01.766817 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.766316 2583 projected.go:196] Error preparing data for projected volume kube-api-access-62lrs for pod openshift-network-diagnostics/network-check-target-vstcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:01.766817 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.766374 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs podName:708926c2-26a2-4888-ae37-a4c157a07e94 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:03.766355797 +0000 UTC m=+5.433838299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-62lrs" (UniqueName: "kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs") pod "network-check-target-vstcw" (UID: "708926c2-26a2-4888-ae37-a4c157a07e94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:01.885519 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.885412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:01.885519 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.885412 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:01.885857 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.885821 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:01.885978 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.885883 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:01.886044 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.885980 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:01.886101 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:01.886046 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:01.893406 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.893377 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" event={"ID":"d757110b-4b02-4857-853b-949b53441428","Type":"ContainerStarted","Data":"d7c9fe47ec8e23118c1da19d2c76d1d5178a1792da5a3c965b8bff3f51b5dc76"} Oct 30 21:37:01.894814 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.894786 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ps2p9" event={"ID":"5aa67807-4df5-4e6f-a4f9-a378593abf94","Type":"ContainerStarted","Data":"bd3fed9922a28fad2449285780c43ced8f2188394a6fabe58ea5fab5f6777a05"} Oct 30 21:37:01.896157 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.896130 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6knhd" event={"ID":"788ef145-30de-43f5-8249-799dbb536b1b","Type":"ContainerStarted","Data":"abf664d805b9cc7669cee12830734f25b3feab54e406903a96c5461903ef7c4f"} Oct 30 21:37:01.897801 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.897776 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fb8rx" event={"ID":"c4d7df9d-6dbd-4b9f-a679-00d8d07db782","Type":"ContainerStarted","Data":"5fb865af1615becd8b256e62b2c5e48e5844497a001664efd1fb5888170ddaeb"} Oct 30 21:37:01.899189 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.899164 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"b75ca48e5a4ea8bdca39ae0aed91953075b66d381845b9ff8b2236b757e79364"} Oct 30 21:37:01.900786 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.900769 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j6hd8" event={"ID":"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc","Type":"ContainerStarted","Data":"039181eb63ea760ad52c55d4bf65f406ccd3f46ba3ed7e7183d0622396ab338c"} Oct 30 21:37:01.902070 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.902049 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftmsf" event={"ID":"9a49810a-da35-4032-b7ac-a6e3c7453d21","Type":"ContainerStarted","Data":"49ab811d4b3a6d0e71909de5e5344f69b2b09fee4400496ef0d5f4fec1e3bf8b"} Oct 30 21:37:01.903360 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.903306 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" event={"ID":"556b7537-074f-46ce-8db5-e1a0a7b94dca","Type":"ContainerStarted","Data":"cf97dafb596cc66eff58442e40d8f56d0468e9994a14f72fb0c3a30b1231b4a0"} Oct 30 21:37:01.904458 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:01.904378 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hw57b" event={"ID":"10088b05-b328-4162-8ef6-39e60f240fb3","Type":"ContainerStarted","Data":"52d00644bd4350dd3c1955975d721d6e21274bf684f44f3acc5e614bf27e4d5a"} Oct 30 21:37:02.180809 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:02.180684 2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:32:00 +0000 UTC" deadline="2028-05-30 15:59:41.574056236 +0000 UTC" Oct 30 21:37:02.180809 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:02.180746 2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13866h22m39.393315148s" Oct 30 21:37:03.687660 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:03.687628 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:03.688260 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:03.687668 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:03.688260 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.687798 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:03.688260 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.687850 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret podName:843e4020-8588-4ead-adda-ead6c9e0882b nodeName:}" failed. No retries permitted until 2026-10-30 21:37:07.687837126 +0000 UTC m=+9.355319639 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret") pod "global-pull-secret-syncer-9rfmn" (UID: "843e4020-8588-4ead-adda-ead6c9e0882b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:03.688260 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.687894 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:03.688260 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.687920 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs podName:9b80443a-852a-4e9e-8a6d-26de01cd2912 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:07.687912263 +0000 UTC m=+9.355394763 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs") pod "network-metrics-daemon-sbp5h" (UID: "9b80443a-852a-4e9e-8a6d-26de01cd2912") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:03.789129 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:03.789086 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:03.789362 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.789341 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:37:03.789520 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.789371 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:37:03.789520 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.789385 2583 projected.go:196] Error preparing data for projected volume kube-api-access-62lrs for pod openshift-network-diagnostics/network-check-target-vstcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:03.789520 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.789453 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs podName:708926c2-26a2-4888-ae37-a4c157a07e94 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:07.789433831 +0000 UTC m=+9.456916351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-62lrs" (UniqueName: "kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs") pod "network-check-target-vstcw" (UID: "708926c2-26a2-4888-ae37-a4c157a07e94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:03.887347 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:03.887210 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:03.887531 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.887409 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:03.887531 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:03.887501 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:03.887668 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.887572 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:03.887668 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:03.887627 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:03.887778 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:03.887717 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:04.931143 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:04.930978 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"3ab1a8fb7b394ed5c2ad3173d5294ab9","Type":"ContainerStarted","Data":"2eec01a4e462ad04d77e4fbdc1e9ede6d5b8ab8d413f8ecafa22ec8d473f31e9"} Oct 30 21:37:04.935733 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:04.935703 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"1d8f9506264df06e876fee4a0052af02","Type":"ContainerDied","Data":"f5217bdad722aafd4341c65a44fc35bf2ed9e33e5e0623f6dce6afcc45e91c62"} Oct 30 21:37:04.936256 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:04.936224 2583 generic.go:356] "Generic (PLEG): container finished" podID="1d8f9506264df06e876fee4a0052af02" containerID="f5217bdad722aafd4341c65a44fc35bf2ed9e33e5e0623f6dce6afcc45e91c62" exitCode=0 Oct 30 21:37:05.194514 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:05.191405 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" podStartSLOduration=5.191388559 podStartE2EDuration="5.191388559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:05.056774913 +0000 UTC m=+6.724257437" watchObservedRunningTime="2026-10-30 21:37:05.191388559 +0000 UTC m=+6.858871095" Oct 30 21:37:05.885334 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:05.885297 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:05.885547 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:05.885460 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:05.885613 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:05.885548 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:05.885658 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:05.885625 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:05.885703 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:05.885692 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:05.885803 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:05.885781 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:05.946319 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:05.946281 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"1d8f9506264df06e876fee4a0052af02","Type":"ContainerStarted","Data":"9fb3d44906776eb889fd1bd8d1a66f05cbcd0680ff184a0711d7f1da997637b4"} Oct 30 21:37:05.949867 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:05.949833 2583 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="b675ba0afc64c8dd9e5d82ecb871fe51cd84744c341f0760f3884bd1d7c3fe41" exitCode=0 Oct 30 21:37:05.950847 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:05.950806 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hw57b" event={"ID":"10088b05-b328-4162-8ef6-39e60f240fb3","Type":"ContainerDied","Data":"b675ba0afc64c8dd9e5d82ecb871fe51cd84744c341f0760f3884bd1d7c3fe41"} Oct 30 21:37:06.075100 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:06.075038 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" podStartSLOduration=6.075018653 podStartE2EDuration="6.075018653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:06.073893796 +0000 UTC m=+7.741376359" watchObservedRunningTime="2026-10-30 21:37:06.075018653 +0000 UTC m=+7.742501180" Oct 30 21:37:06.961892 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:06.961812 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6knhd" event={"ID":"788ef145-30de-43f5-8249-799dbb536b1b","Type":"ContainerStarted","Data":"6da630173b90809b6b7e794d6f5bb45258fe2ef6f1da30dd83f0b5610ef4b18d"} Oct 30 21:37:07.090964 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:07.090911 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6knhd" podStartSLOduration=2.133294304 podStartE2EDuration="7.090895763s" totalImagesPullingTime="4.957601459s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:07.090265792 +0000 UTC m=+8.757748316" watchObservedRunningTime="2026-10-30 21:37:07.090895763 +0000 UTC m=+8.758378304" Oct 30 21:37:07.720464 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:07.720427 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:07.720680 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:07.720479 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:07.720780 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.720753 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:07.720883 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.720816 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret podName:843e4020-8588-4ead-adda-ead6c9e0882b nodeName:}" failed. No retries permitted until 2026-10-30 21:37:15.72079879 +0000 UTC m=+17.388281294 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret") pod "global-pull-secret-syncer-9rfmn" (UID: "843e4020-8588-4ead-adda-ead6c9e0882b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:07.720957 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.720884 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:07.720957 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.720923 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs podName:9b80443a-852a-4e9e-8a6d-26de01cd2912 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:15.720910292 +0000 UTC m=+17.388392808 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs") pod "network-metrics-daemon-sbp5h" (UID: "9b80443a-852a-4e9e-8a6d-26de01cd2912") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:07.821697 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:07.821299 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:07.821697 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.821513 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:37:07.821697 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.821537 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:37:07.821697 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.821550 2583 projected.go:196] Error preparing data for projected volume kube-api-access-62lrs for pod openshift-network-diagnostics/network-check-target-vstcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:07.821697 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.821613 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs podName:708926c2-26a2-4888-ae37-a4c157a07e94 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:15.821595226 +0000 UTC m=+17.489077748 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-62lrs" (UniqueName: "kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs") pod "network-check-target-vstcw" (UID: "708926c2-26a2-4888-ae37-a4c157a07e94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:07.886269 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:07.885803 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:07.886269 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.885923 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:07.886269 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:07.885974 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:07.886269 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.886035 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:07.886269 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:07.886067 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:07.886269 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:07.886116 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:07.970070 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:07.969886 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftmsf" event={"ID":"9a49810a-da35-4032-b7ac-a6e3c7453d21","Type":"ContainerStarted","Data":"1454d83ccd818a749a8777ab5c6eaecc07f135359b1b4161bb079e265fd88264"} Oct 30 21:37:08.977226 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:08.977173 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" event={"ID":"556b7537-074f-46ce-8db5-e1a0a7b94dca","Type":"ContainerStarted","Data":"dd9bfeefe07da5bcc36c55a9f933a8f2c9a956fd73abe364f822c72a2c1043d2"} Oct 30 21:37:09.327989 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:09.327892 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:09.885954 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:09.885918 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:09.886145 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:09.886068 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:09.886546 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:09.886520 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:09.886941 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:09.886709 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:09.886941 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:09.886789 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:09.886941 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:09.886870 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:10.710922 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:10.710895 2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:37:10.796312 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:10.796200 2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:37:10.710915114Z","UUID":"903c73cf-5a1a-4af1-8729-4536145cc125","Handler":null,"Name":"","Endpoint":""} Oct 30 21:37:10.798253 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:10.798227 2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:37:10.798253 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:10.798260 2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:37:10.983477 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:10.983412 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" event={"ID":"556b7537-074f-46ce-8db5-e1a0a7b94dca","Type":"ContainerStarted","Data":"110a51cb2b22eec0a0264be14d6a717727d3ac50b585137b81d08b0d8a067e4e"} Oct 30 21:37:10.985808 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:10.985776 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-ps2p9" event={"ID":"5aa67807-4df5-4e6f-a4f9-a378593abf94","Type":"ContainerStarted","Data":"6e4660e5c8dad47c38a7738c80e02a97638502013c2e48e79ae2af86fc7bf59b"} Oct 30 21:37:10.988197 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:10.988167 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-j6hd8" event={"ID":"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc","Type":"ContainerStarted","Data":"229e7eb0a912cfbc7973a4db283351d9cb38f361f8a0a2942ac3554640c8a902"} Oct 30 21:37:11.112667 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.112619 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ftmsf" podStartSLOduration=5.782999652 podStartE2EDuration="12.112602384s" totalImagesPullingTime="6.329602732s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:08.100820906 +0000 UTC m=+9.768303428" watchObservedRunningTime="2026-10-30 21:37:11.112602384 +0000 UTC m=+12.780084908" Oct 30 21:37:11.113335 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.113258 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-ps2p9" podStartSLOduration=2.260379775 podStartE2EDuration="11.11324304s" totalImagesPullingTime="8.852863265s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:11.112325462 +0000 UTC m=+12.779808058" watchObservedRunningTime="2026-10-30 21:37:11.11324304 +0000 UTC m=+12.780725566" Oct 30 21:37:11.197854 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.197821 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:11.198383 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.198360 2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:11.374933 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.373844 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-j6hd8" podStartSLOduration=2.664021316 podStartE2EDuration="11.373819894s" totalImagesPullingTime="8.709798578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:11.24632266 +0000 UTC m=+12.913805182" watchObservedRunningTime="2026-10-30 21:37:11.373819894 +0000 UTC m=+13.041302419" Oct 30 21:37:11.885600 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.885567 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:11.886037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.885602 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:11.886037 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:11.885700 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:11.886037 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:11.885772 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:11.886037 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.885822 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:11.886037 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:11.885925 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:11.992377 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.992265 2583 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="00f5ada52b43c2ca91fdbbde9f1a62508ec01d5a8d79b0db9737a3a4f785c390" exitCode=0 Oct 30 21:37:11.992377 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.992354 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hw57b" event={"ID":"10088b05-b328-4162-8ef6-39e60f240fb3","Type":"ContainerDied","Data":"00f5ada52b43c2ca91fdbbde9f1a62508ec01d5a8d79b0db9737a3a4f785c390"} Oct 30 21:37:11.993718 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:11.993702 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:37:12.996866 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:12.996819 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" event={"ID":"556b7537-074f-46ce-8db5-e1a0a7b94dca","Type":"ContainerStarted","Data":"f93c628621c8be5fecf1b4f29301c8aeb96f6418cf0744b0b81454beb6652a9f"} Oct 30 21:37:12.998476 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:12.998445 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" event={"ID":"d757110b-4b02-4857-853b-949b53441428","Type":"ContainerStarted","Data":"0793fdd0a19a8b72e20614c8bf4e5d03c0c5460a95e2add0861ca02db99696d4"} Oct 30 21:37:12.999973 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:12.999942 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fb8rx" event={"ID":"c4d7df9d-6dbd-4b9f-a679-00d8d07db782","Type":"ContainerStarted","Data":"0412d092fa16522133447b886b26378a655e872218810256bede1fb8fa98da3b"} Oct 30 21:37:13.117155 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:13.117108 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" podStartSLOduration=2.384913711 podStartE2EDuration="13.117093033s" totalImagesPullingTime="10.732179322s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:13.116847547 +0000 UTC m=+14.784330078" watchObservedRunningTime="2026-10-30 21:37:13.117093033 +0000 UTC m=+14.784575554" Oct 30 21:37:13.237930 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:13.237882 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fb8rx" podStartSLOduration=3.912401027 podStartE2EDuration="13.237866585s" totalImagesPullingTime="9.325465558s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:13.237550529 +0000 UTC m=+14.905033073" watchObservedRunningTime="2026-10-30 21:37:13.237866585 +0000 UTC m=+14.905349106" Oct 30 21:37:13.358766 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:13.358665 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" podStartSLOduration=2.364141102 podStartE2EDuration="13.358646772s" totalImagesPullingTime="10.99450567s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:13.358378648 +0000 UTC m=+15.025861170" watchObservedRunningTime="2026-10-30 21:37:13.358646772 +0000 UTC m=+15.026129293" Oct 30 21:37:13.885218 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:13.885177 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:13.885398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:13.885234 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:13.885398 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:13.885336 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:13.885553 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:13.885399 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:13.885553 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:13.885544 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:13.885670 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:13.885646 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:14.004991 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:14.004954 2583 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="bc32484bfe7c64fc4b8cd34caa323e1edec9e7a98a8b5eefcf7c894feafcf956" exitCode=0 Oct 30 21:37:14.005555 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:14.005027 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hw57b" event={"ID":"10088b05-b328-4162-8ef6-39e60f240fb3","Type":"ContainerDied","Data":"bc32484bfe7c64fc4b8cd34caa323e1edec9e7a98a8b5eefcf7c894feafcf956"} Oct 30 21:37:15.786020 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:15.785808 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:15.786539 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:15.786038 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:15.786539 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.785966 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:15.786539 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.786164 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:15.786539 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.786164 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs podName:9b80443a-852a-4e9e-8a6d-26de01cd2912 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:31.786145192 +0000 UTC m=+33.453627709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs") pod "network-metrics-daemon-sbp5h" (UID: "9b80443a-852a-4e9e-8a6d-26de01cd2912") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:15.786539 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.786208 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret podName:843e4020-8588-4ead-adda-ead6c9e0882b nodeName:}" failed. No retries permitted until 2026-10-30 21:37:31.786197171 +0000 UTC m=+33.453679670 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret") pod "global-pull-secret-syncer-9rfmn" (UID: "843e4020-8588-4ead-adda-ead6c9e0882b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:15.885898 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:15.885865 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:15.885898 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:15.885907 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:15.886118 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:15.885928 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:15.886118 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.886043 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:15.886190 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.886138 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:15.886239 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.886219 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:15.886405 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:15.886384 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:15.886555 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.886538 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 21:37:15.886599 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.886561 2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 21:37:15.886599 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.886573 2583 projected.go:196] Error preparing data for projected volume kube-api-access-62lrs for pod openshift-network-diagnostics/network-check-target-vstcw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:15.886677 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:15.886628 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs podName:708926c2-26a2-4888-ae37-a4c157a07e94 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:31.886611607 +0000 UTC m=+33.554094109 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-62lrs" (UniqueName: "kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs") pod "network-check-target-vstcw" (UID: "708926c2-26a2-4888-ae37-a4c157a07e94") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 21:37:16.012038 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:16.012003 2583 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="82b6da202d537685cffde05ec06d7eb4c299c95de50d8056724c4189938e57fe" exitCode=0 Oct 30 21:37:16.012192 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:16.012048 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hw57b" event={"ID":"10088b05-b328-4162-8ef6-39e60f240fb3","Type":"ContainerDied","Data":"82b6da202d537685cffde05ec06d7eb4c299c95de50d8056724c4189938e57fe"} Oct 30 21:37:17.017952 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.017764 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"8980f531d8cf59ad4ab5b97ef5113bfdb407b8fcb13374db84d5df8d1092d928"} Oct 30 21:37:17.018434 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.018014 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"f8a57f10b76ddad62c4be341f43341b932bc7e940d03f1dd5714c011afa8362e"} Oct 30 21:37:17.018434 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.018065 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"f8a3d5f44684322019ca9ac263318891fbda1085c835cabc2b62879052dad4a8"} Oct 30 21:37:17.018434 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.018116 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"faf0a11861112210ce334a108dc3514ff39f2e8105c2a64d563758aa216ccba3"} Oct 30 21:37:17.018434 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.018158 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"a31078ec7cdcf376678e6781798021c2c4ef78cfbf9db5187103c9ccebce47e4"} Oct 30 21:37:17.018434 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.018199 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"8c4e41b5a0d0e74c21f7f5313f8e51e2e0f9ff10aeaa788226f75837b28738e0"} Oct 30 21:37:17.885125 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.885078 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:17.885289 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.885095 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:17.885289 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:17.885200 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:17.885289 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:17.885207 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:17.885289 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:17.885285 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:17.885499 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:17.885371 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:19.024918 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:19.024879 2583 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="fc108a82d389a5c0587529204efc6e21b790206197f2d19a10b684cc0abca5f1" exitCode=0 Oct 30 21:37:19.024918 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:19.024922 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hw57b" event={"ID":"10088b05-b328-4162-8ef6-39e60f240fb3","Type":"ContainerDied","Data":"fc108a82d389a5c0587529204efc6e21b790206197f2d19a10b684cc0abca5f1"} Oct 30 21:37:19.885230 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:19.885190 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:19.885403 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:19.885269 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:19.885403 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:19.885359 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:19.885403 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:19.885367 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:19.885541 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:19.885469 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:19.885575 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:19.885539 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:20.030507 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:20.030473 2583 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="1eb9e3ccd035eb9115e69d68551f7878c8bd8c4e19db533dddc88c5c781dd2a8" exitCode=0 Oct 30 21:37:20.030943 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:20.030519 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hw57b" event={"ID":"10088b05-b328-4162-8ef6-39e60f240fb3","Type":"ContainerDied","Data":"1eb9e3ccd035eb9115e69d68551f7878c8bd8c4e19db533dddc88c5c781dd2a8"} Oct 30 21:37:20.033902 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:20.033880 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"71f6797d39c338bdfa8a1e338a03062040a71521ce3b1a2a3f2cb57c8f7e0b32"} Oct 30 21:37:21.039346 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:21.039311 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-hw57b" event={"ID":"10088b05-b328-4162-8ef6-39e60f240fb3","Type":"ContainerStarted","Data":"d574b4dff1c5448c1445d2f70169e712552ba12c72576bff56d5878fc7974948"} Oct 30 21:37:21.264963 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:21.264909 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-hw57b" podStartSLOduration=8.489687955 podStartE2EDuration="22.264893993s" totalImagesPullingTime="13.775206038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:21.264339911 +0000 UTC m=+22.931822431" watchObservedRunningTime="2026-10-30 21:37:21.264893993 +0000 UTC m=+22.932376513" Oct 30 21:37:21.885493 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:21.885302 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:21.885782 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:21.885348 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:21.885782 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:21.885577 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:21.885782 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:21.885357 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:21.885782 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:21.885658 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:21.885782 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:21.885725 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:22.045052 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:22.045019 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" event={"ID":"eab1dc97-e877-4b0f-ab0c-406eb137b296","Type":"ContainerStarted","Data":"3a4e41042146558ed5f95c1c95844377cfe46fdafdb7d7aa22e02a680fde2246"} Oct 30 21:37:22.328533 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:22.328485 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" podStartSLOduration=8.064056457 podStartE2EDuration="23.328471134s" totalImagesPullingTime="15.264414677s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:22.277705646 +0000 UTC m=+23.945188166" watchObservedRunningTime="2026-10-30 21:37:22.328471134 +0000 UTC m=+23.995953654" Oct 30 21:37:23.047230 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:23.047196 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:23.047687 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:23.047281 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:23.047687 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:23.047324 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:23.062832 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:23.062810 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:23.062984 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:23.062908 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:37:23.885401 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:23.885323 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:23.885547 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:23.885343 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:23.885547 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:23.885469 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:23.885547 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:23.885512 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:23.885671 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:23.885593 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:23.885717 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:23.885672 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:24.018986 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:24.018943 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rfmn"] Oct 30 21:37:24.021983 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:24.021955 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vstcw"] Oct 30 21:37:24.022470 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:24.022453 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sbp5h"] Oct 30 21:37:24.049038 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:24.049014 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:24.049038 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:24.049036 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:24.049512 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:24.049025 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:24.049512 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:24.049112 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:24.049576 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:24.049531 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:24.049636 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:24.049620 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:25.885357 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:25.885329 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:25.885712 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:25.885407 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:25.885712 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:25.885436 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:25.885712 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:25.885513 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:25.885712 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:25.885576 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:25.885712 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:25.885636 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:27.885626 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:27.885597 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:27.885996 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:27.885700 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:27.885996 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:27.885712 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:27.885996 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:27.885735 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:27.885996 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:27.885819 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:27.885996 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:27.885874 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:29.885874 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:29.885692 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:29.886361 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:29.885719 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:29.886361 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:29.885735 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:29.886361 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:29.885970 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sbp5h" podUID="9b80443a-852a-4e9e-8a6d-26de01cd2912" Oct 30 21:37:29.886361 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:29.886028 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vstcw" podUID="708926c2-26a2-4888-ae37-a4c157a07e94" Oct 30 21:37:29.886361 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:29.886101 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-9rfmn" podUID="843e4020-8588-4ead-adda-ead6c9e0882b" Oct 30 21:37:30.095259 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.095233 2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeReady" Oct 30 21:37:30.095430 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.095350 2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 21:37:30.219435 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.219346 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xfjrb"] Oct 30 21:37:30.221953 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.221934 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5gfv2"] Oct 30 21:37:30.222081 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.222066 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.223966 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.223946 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:37:30.278217 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.278183 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:37:30.278217 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.278177 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xfjrb"] Oct 30 21:37:30.278497 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.278290 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:37:30.278497 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.278318 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb4gr\"" Oct 30 21:37:30.278497 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.278361 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-phtlh\"" Oct 30 21:37:30.279740 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.279718 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:37:30.280003 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.279990 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:37:30.280357 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.280341 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:37:30.286012 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.285994 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5gfv2"] Oct 30 21:37:30.288374 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.288352 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sz7b\" (UniqueName: \"kubernetes.io/projected/261c8437-5164-4544-bd94-90706f1d919b-kube-api-access-2sz7b\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:37:30.288490 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.288381 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c8437-5164-4544-bd94-90706f1d919b-cert\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:37:30.288490 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.288398 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec88c90e-fce8-4f91-9d30-e554174ef776-tmp-dir\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.288490 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.288430 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec88c90e-fce8-4f91-9d30-e554174ef776-config-volume\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.288490 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.288446 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.288690 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.288624 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzrdr\" (UniqueName: \"kubernetes.io/projected/ec88c90e-fce8-4f91-9d30-e554174ef776-kube-api-access-mzrdr\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.389591 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.389553 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sz7b\" (UniqueName: \"kubernetes.io/projected/261c8437-5164-4544-bd94-90706f1d919b-kube-api-access-2sz7b\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:37:30.389591 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.389592 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c8437-5164-4544-bd94-90706f1d919b-cert\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:37:30.389838 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.389611 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec88c90e-fce8-4f91-9d30-e554174ef776-tmp-dir\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.389838 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.389799 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec88c90e-fce8-4f91-9d30-e554174ef776-config-volume\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.389934 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.389842 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.389934 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.389895 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrdr\" (UniqueName: \"kubernetes.io/projected/ec88c90e-fce8-4f91-9d30-e554174ef776-kube-api-access-mzrdr\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.390039 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:30.390020 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:37:30.390101 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:30.390090 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls podName:ec88c90e-fce8-4f91-9d30-e554174ef776 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:30.890071147 +0000 UTC m=+32.557553664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls") pod "dns-default-xfjrb" (UID: "ec88c90e-fce8-4f91-9d30-e554174ef776") : secret "dns-default-metrics-tls" not found Oct 30 21:37:30.390176 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.390159 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec88c90e-fce8-4f91-9d30-e554174ef776-tmp-dir\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.390448 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.390404 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec88c90e-fce8-4f91-9d30-e554174ef776-config-volume\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.393775 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.393756 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c8437-5164-4544-bd94-90706f1d919b-cert\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:37:30.453867 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.453836 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrdr\" (UniqueName: \"kubernetes.io/projected/ec88c90e-fce8-4f91-9d30-e554174ef776-kube-api-access-mzrdr\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.454040 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.454021 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sz7b\" (UniqueName: \"kubernetes.io/projected/261c8437-5164-4544-bd94-90706f1d919b-kube-api-access-2sz7b\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:37:30.538758 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.538718 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:37:30.893446 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.893348 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:30.893827 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:30.893506 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:37:30.893827 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:30.893565 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls podName:ec88c90e-fce8-4f91-9d30-e554174ef776 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:31.893552252 +0000 UTC m=+33.561034770 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls") pod "dns-default-xfjrb" (UID: "ec88c90e-fce8-4f91-9d30-e554174ef776") : secret "dns-default-metrics-tls" not found Oct 30 21:37:30.919659 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:30.919632 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5gfv2"] Oct 30 21:37:30.919887 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:30.919868 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261c8437_5164_4544_bd94_90706f1d919b.slice/crio-7b19a7a890b1bf836805baa7d3ec840b7c58fbd334a7fa7bf067c20470e711f3 WatchSource:0}: Error finding container 7b19a7a890b1bf836805baa7d3ec840b7c58fbd334a7fa7bf067c20470e711f3: Status 404 returned error can't find the container with id 7b19a7a890b1bf836805baa7d3ec840b7c58fbd334a7fa7bf067c20470e711f3 Oct 30 21:37:31.065210 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.065174 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5gfv2" event={"ID":"261c8437-5164-4544-bd94-90706f1d919b","Type":"ContainerStarted","Data":"7b19a7a890b1bf836805baa7d3ec840b7c58fbd334a7fa7bf067c20470e711f3"} Oct 30 21:37:31.798924 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.798879 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:31.798924 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.798918 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:31.799118 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:31.799040 2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:31.799118 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:31.799044 2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:31.799118 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:31.799101 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret podName:843e4020-8588-4ead-adda-ead6c9e0882b nodeName:}" failed. No retries permitted until 2026-10-30 21:38:03.799088887 +0000 UTC m=+65.466571386 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret") pod "global-pull-secret-syncer-9rfmn" (UID: "843e4020-8588-4ead-adda-ead6c9e0882b") : object "kube-system"/"original-pull-secret" not registered Oct 30 21:37:31.799118 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:31.799114 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs podName:9b80443a-852a-4e9e-8a6d-26de01cd2912 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:03.79910816 +0000 UTC m=+65.466590660 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs") pod "network-metrics-daemon-sbp5h" (UID: "9b80443a-852a-4e9e-8a6d-26de01cd2912") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 21:37:31.885552 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.885519 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:31.885552 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.885542 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:37:31.885745 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.885728 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:37:31.899193 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.899173 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:31.899538 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.899216 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:31.899538 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:31.899339 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:37:31.899538 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:31.899410 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls podName:ec88c90e-fce8-4f91-9d30-e554174ef776 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:33.899389397 +0000 UTC m=+35.566871909 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls") pod "dns-default-xfjrb" (UID: "ec88c90e-fce8-4f91-9d30-e554174ef776") : secret "dns-default-metrics-tls" not found Oct 30 21:37:31.939735 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.939711 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:37:31.990530 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.990507 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:37:31.990657 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.990534 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:37:31.990657 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.990565 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v9xch\"" Oct 30 21:37:31.990657 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.990566 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:37:31.990657 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.990629 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ktl9g\"" Oct 30 21:37:31.990809 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:31.990692 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7gkt\"" Oct 30 21:37:32.002465 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:32.002437 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:32.197560 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:32.197514 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:32.560517 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:32.560486 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vstcw"] Oct 30 21:37:32.560779 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:32.560752 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod708926c2_26a2_4888_ae37_a4c157a07e94.slice/crio-14d1576a8ab84ef928bf6bb26d15c977b813ce17774d2237c745a34d8f499ddb WatchSource:0}: Error finding container 14d1576a8ab84ef928bf6bb26d15c977b813ce17774d2237c745a34d8f499ddb: Status 404 returned error can't find the container with id 14d1576a8ab84ef928bf6bb26d15c977b813ce17774d2237c745a34d8f499ddb Oct 30 21:37:33.070405 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.070376 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vstcw" event={"ID":"708926c2-26a2-4888-ae37-a4c157a07e94","Type":"ContainerStarted","Data":"14d1576a8ab84ef928bf6bb26d15c977b813ce17774d2237c745a34d8f499ddb"} Oct 30 21:37:33.689500 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.689467 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fcsg4"] Oct 30 21:37:33.692205 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.692183 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.749907 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.749867 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:37:33.750070 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.749996 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:37:33.750121 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.750089 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:37:33.802034 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.802006 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fcsg4"] Oct 30 21:37:33.812374 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.812342 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42c37929-ba43-4002-877d-a3ab2f5723b8-data-volume\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.812374 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.812374 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.812610 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.812394 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr849\" (UniqueName: \"kubernetes.io/projected/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-api-access-kr849\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.812610 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.812544 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42c37929-ba43-4002-877d-a3ab2f5723b8-crio-socket\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.812610 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.812583 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.833330 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.833303 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:37:33.833494 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.833334 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlcqp\"" Oct 30 21:37:33.913446 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.913396 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:33.913581 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.913490 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42c37929-ba43-4002-877d-a3ab2f5723b8-crio-socket\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.913581 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.913518 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.913581 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.913541 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42c37929-ba43-4002-877d-a3ab2f5723b8-data-volume\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.913581 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.913565 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.913712 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:33.913576 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:37:33.913712 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.913587 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr849\" (UniqueName: \"kubernetes.io/projected/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-api-access-kr849\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.913712 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:33.913633 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls podName:ec88c90e-fce8-4f91-9d30-e554174ef776 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:37.913618173 +0000 UTC m=+39.581100671 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls") pod "dns-default-xfjrb" (UID: "ec88c90e-fce8-4f91-9d30-e554174ef776") : secret "dns-default-metrics-tls" not found Oct 30 21:37:33.913886 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:33.913831 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:37:33.913886 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:33.913874 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:34.413862897 +0000 UTC m=+36.081345415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : configmap "kube-rbac-proxy" not found Oct 30 21:37:33.914003 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.913978 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42c37929-ba43-4002-877d-a3ab2f5723b8-crio-socket\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.914115 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:33.914052 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:37:33.914115 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:33.914089 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:34.414076369 +0000 UTC m=+36.081558872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : secret "insights-runtime-extractor-tls" not found Oct 30 21:37:33.914256 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.914176 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42c37929-ba43-4002-877d-a3ab2f5723b8-data-volume\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:33.975554 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:33.975521 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr849\" (UniqueName: \"kubernetes.io/projected/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-api-access-kr849\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:34.417224 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:34.417187 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:34.417690 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:34.417295 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:34.417690 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:34.417357 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:37:34.417690 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:34.417399 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:37:34.417690 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:34.417463 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:35.41744173 +0000 UTC m=+37.084924246 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : secret "insights-runtime-extractor-tls" not found Oct 30 21:37:34.417690 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:34.417482 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:35.417475147 +0000 UTC m=+37.084957646 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : configmap "kube-rbac-proxy" not found Oct 30 21:37:34.578192 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:34.578139 2583 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage4050041068/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T213733Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=d3637cf20e69865889c8f08d9ef4aa694bb58ec85b43435197c6f2197446147c®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793397153~hmac=e35c1a0d0524b0b93db9ebf2efbfd03b897eabf19a4dc53cdd10c63dec73a08c\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab" Oct 30 21:37:34.578459 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:34.578403 2583 kuberuntime_manager.go:1707] "Unhandled Error" err="container serve-healthcheck-canary start failed in pod ingress-canary-5gfv2_openshift-ingress-canary(261c8437-5164-4544-bd94-90706f1d919b): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage4050041068/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T213733Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=d3637cf20e69865889c8f08d9ef4aa694bb58ec85b43435197c6f2197446147c®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793397153~hmac=e35c1a0d0524b0b93db9ebf2efbfd03b897eabf19a4dc53cdd10c63dec73a08c\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError" Oct 30 21:37:34.578579 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:34.578496 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage4050041068/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T213733Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=d3637cf20e69865889c8f08d9ef4aa694bb58ec85b43435197c6f2197446147c®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793397153~hmac=e35c1a0d0524b0b93db9ebf2efbfd03b897eabf19a4dc53cdd10c63dec73a08c\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-5gfv2" podUID="261c8437-5164-4544-bd94-90706f1d919b" Oct 30 21:37:35.077367 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:35.077309 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage4050041068/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T213733Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=d3637cf20e69865889c8f08d9ef4aa694bb58ec85b43435197c6f2197446147c®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793397153~hmac=e35c1a0d0524b0b93db9ebf2efbfd03b897eabf19a4dc53cdd10c63dec73a08c\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-5gfv2" podUID="261c8437-5164-4544-bd94-90706f1d919b" Oct 30 21:37:35.294759 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.294721 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz"] Oct 30 21:37:35.298094 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.298072 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" Oct 30 21:37:35.353162 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.353090 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz"] Oct 30 21:37:35.415904 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.415869 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2fnrw\"" Oct 30 21:37:35.424984 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.424957 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nzqb\" (UniqueName: \"kubernetes.io/projected/114097ab-0272-43e5-a492-f12db369977a-kube-api-access-8nzqb\") pod \"migrator-66df5d496b-4x5sz\" (UID: \"114097ab-0272-43e5-a492-f12db369977a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" Oct 30 21:37:35.425374 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.424997 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:35.425374 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.425082 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:35.425374 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:35.425124 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:37:35.425374 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:35.425183 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:37.425166267 +0000 UTC m=+39.092648769 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : configmap "kube-rbac-proxy" not found Oct 30 21:37:35.425374 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:35.425216 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:37:35.425374 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:35.425257 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:37.425245989 +0000 UTC m=+39.092728504 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : secret "insights-runtime-extractor-tls" not found Oct 30 21:37:35.440951 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.440930 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:37:35.441327 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.441308 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:37:35.526474 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.526436 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzqb\" (UniqueName: \"kubernetes.io/projected/114097ab-0272-43e5-a492-f12db369977a-kube-api-access-8nzqb\") pod \"migrator-66df5d496b-4x5sz\" (UID: \"114097ab-0272-43e5-a492-f12db369977a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" Oct 30 21:37:35.588831 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.588790 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzqb\" (UniqueName: \"kubernetes.io/projected/114097ab-0272-43e5-a492-f12db369977a-kube-api-access-8nzqb\") pod \"migrator-66df5d496b-4x5sz\" (UID: \"114097ab-0272-43e5-a492-f12db369977a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" Oct 30 21:37:35.610236 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.610162 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" Oct 30 21:37:35.979101 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:37:35.979072 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114097ab_0272_43e5_a492_f12db369977a.slice/crio-3b07d769b1a4d468ce46a90757f89336ff03998c61ff7b572093f161c30fee89 WatchSource:0}: Error finding container 3b07d769b1a4d468ce46a90757f89336ff03998c61ff7b572093f161c30fee89: Status 404 returned error can't find the container with id 3b07d769b1a4d468ce46a90757f89336ff03998c61ff7b572093f161c30fee89 Oct 30 21:37:35.979366 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:35.979346 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz"] Oct 30 21:37:36.079610 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:36.079572 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vstcw" event={"ID":"708926c2-26a2-4888-ae37-a4c157a07e94","Type":"ContainerStarted","Data":"abd109b19a6142368d863188c506b1db87796c3005b61de089c3490932d87741"} Oct 30 21:37:36.079862 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:36.079833 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:37:36.080636 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:36.080615 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" event={"ID":"114097ab-0272-43e5-a492-f12db369977a","Type":"ContainerStarted","Data":"3b07d769b1a4d468ce46a90757f89336ff03998c61ff7b572093f161c30fee89"} Oct 30 21:37:36.199096 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:36.199037 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vstcw" podStartSLOduration=33.976428096 podStartE2EDuration="37.199024636s" totalImagesPullingTime="3.22259654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:36.198923204 +0000 UTC m=+37.866405726" watchObservedRunningTime="2026-10-30 21:37:36.199024636 +0000 UTC m=+37.866507157" Oct 30 21:37:37.440095 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:37.440056 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:37.440610 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:37.440102 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:37.440610 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:37.440176 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:37:37.440610 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:37.440196 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:37:37.440610 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:37.440243 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:41.440230796 +0000 UTC m=+43.107713298 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : secret "insights-runtime-extractor-tls" not found Oct 30 21:37:37.440610 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:37.440257 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:41.440250944 +0000 UTC m=+43.107733442 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : configmap "kube-rbac-proxy" not found Oct 30 21:37:37.943532 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:37.943501 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:37.943694 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:37.943628 2583 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 21:37:37.943694 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:37.943682 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls podName:ec88c90e-fce8-4f91-9d30-e554174ef776 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:45.943668524 +0000 UTC m=+47.611151023 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls") pod "dns-default-xfjrb" (UID: "ec88c90e-fce8-4f91-9d30-e554174ef776") : secret "dns-default-metrics-tls" not found Oct 30 21:37:38.086260 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:38.086215 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" event={"ID":"114097ab-0272-43e5-a492-f12db369977a","Type":"ContainerStarted","Data":"3e817cbc9204a3afbc3a51ed04000df0c110659c126317cf679ee85d45a78b58"} Oct 30 21:37:38.216514 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:38.216460 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" podStartSLOduration=1.458900361 podStartE2EDuration="3.216445709s" totalImagesPullingTime="1.757545348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:38.215300384 +0000 UTC m=+39.882782905" watchObservedRunningTime="2026-10-30 21:37:38.216445709 +0000 UTC m=+39.883928231" Oct 30 21:37:41.468175 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:41.468132 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:41.468175 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:41.468184 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:41.468754 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:41.468255 2583 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 21:37:41.468754 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:41.468310 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:37:41.468754 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:41.468340 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:49.468315179 +0000 UTC m=+51.135797701 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : configmap "kube-rbac-proxy" not found Oct 30 21:37:41.468754 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:41.468360 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:49.468350454 +0000 UTC m=+51.135832962 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : secret "insights-runtime-extractor-tls" not found Oct 30 21:37:41.618963 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:41.618936 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-ps2p9_5aa67807-4df5-4e6f-a4f9-a378593abf94/dns-node-resolver/0.log" Oct 30 21:37:42.414519 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:42.414492 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ftmsf_9a49810a-da35-4032-b7ac-a6e3c7453d21/node-ca/0.log" Oct 30 21:37:43.613468 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:43.613442 2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-4x5sz_114097ab-0272-43e5-a492-f12db369977a/migrator/0.log" Oct 30 21:37:46.004630 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:46.004586 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:46.006817 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:46.006798 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:46.135257 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:46.135221 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:46.584964 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:46.584940 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xfjrb"] Oct 30 21:37:47.109991 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:47.109953 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfjrb" event={"ID":"ec88c90e-fce8-4f91-9d30-e554174ef776","Type":"ContainerStarted","Data":"37640b19f9282865b2bc93be6ab37f4497e2933dd4d16490d11ab717d26138a1"} Oct 30 21:37:49.534274 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:49.534227 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:49.534274 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:49.534267 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:49.534918 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:49.534382 2583 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 21:37:49.534918 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:49.534445 2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls podName:42c37929-ba43-4002-877d-a3ab2f5723b8 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:05.534433506 +0000 UTC m=+67.201916005 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fcsg4" (UID: "42c37929-ba43-4002-877d-a3ab2f5723b8") : secret "insights-runtime-extractor-tls" not found Oct 30 21:37:49.535010 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:49.534917 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:37:50.117730 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:50.117685 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfjrb" event={"ID":"ec88c90e-fce8-4f91-9d30-e554174ef776","Type":"ContainerStarted","Data":"925d4710419202e2f999a6ff2ed77aabc961154306f500113ddb53ae058a1464"} Oct 30 21:37:50.117905 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:50.117749 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xfjrb" event={"ID":"ec88c90e-fce8-4f91-9d30-e554174ef776","Type":"ContainerStarted","Data":"414eca03288aff317d6865e60bf3a4590d08c98ed30f2b85e76be0aa8edb7e37"} Oct 30 21:37:50.117905 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:50.117820 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xfjrb" Oct 30 21:37:50.254110 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:50.254056 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-xfjrb" podStartSLOduration=17.637991402 podStartE2EDuration="20.254043215s" totalImagesPullingTime="2.616051813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:50.253322292 +0000 UTC m=+51.920804824" watchObservedRunningTime="2026-10-30 21:37:50.254043215 +0000 UTC m=+51.921525736" Oct 30 21:37:51.570774 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:51.570725 2583 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage2702461699/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T213750Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=7926d18e553bd704b6b8164bab915c7df52cbd62373ea516ffc75877f5204ed2®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793397170~hmac=b0d648ba2205cf332e20c2e0d1e20fe78ff6b5541719a4f9624d9413f6bd748b\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab" Oct 30 21:37:51.571263 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:51.570948 2583 kuberuntime_manager.go:1707] "Unhandled Error" err="container serve-healthcheck-canary start failed in pod ingress-canary-5gfv2_openshift-ingress-canary(261c8437-5164-4544-bd94-90706f1d919b): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage2702461699/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T213750Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=7926d18e553bd704b6b8164bab915c7df52cbd62373ea516ffc75877f5204ed2®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793397170~hmac=b0d648ba2205cf332e20c2e0d1e20fe78ff6b5541719a4f9624d9413f6bd748b\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError" Oct 30 21:37:51.571263 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:37:51.570997 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage2702461699/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T213750Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=7926d18e553bd704b6b8164bab915c7df52cbd62373ea516ffc75877f5204ed2®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793397170~hmac=b0d648ba2205cf332e20c2e0d1e20fe78ff6b5541719a4f9624d9413f6bd748b\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-5gfv2" podUID="261c8437-5164-4544-bd94-90706f1d919b" Oct 30 21:37:55.061665 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:37:55.061636 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:00.083534 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.083503 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-tvhwl"] Oct 30 21:38:00.087473 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.087452 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:00.122246 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.122223 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:00.156039 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.156011 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:38:00.156463 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.156446 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:38:00.206240 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.206210 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-29zp4\"" Oct 30 21:38:00.207061 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.207044 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7"] Oct 30 21:38:00.209166 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.209149 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tvhwl"] Oct 30 21:38:00.209255 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.209239 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:00.209364 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.209345 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5230b5e1-f7df-4403-adbb-904ef729b5ab-tmp\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:00.209466 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.209431 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpg7q\" (UniqueName: \"kubernetes.io/projected/5230b5e1-f7df-4403-adbb-904ef729b5ab-kube-api-access-hpg7q\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:00.258517 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.258490 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7"] Oct 30 21:38:00.269918 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.269898 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6f8c95d677-txjhg"] Oct 30 21:38:00.271950 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.271937 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.274629 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.274604 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:38:00.274769 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.274713 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-g7wm5\"" Oct 30 21:38:00.310075 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.310050 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a776136-2e97-427d-ad80-4b990b59d897-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qrpk7\" (UID: \"9a776136-2e97-427d-ad80-4b990b59d897\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:00.310181 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.310098 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5230b5e1-f7df-4403-adbb-904ef729b5ab-tmp\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:00.310241 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.310208 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpg7q\" (UniqueName: \"kubernetes.io/projected/5230b5e1-f7df-4403-adbb-904ef729b5ab-kube-api-access-hpg7q\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:00.310450 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.310433 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5230b5e1-f7df-4403-adbb-904ef729b5ab-tmp\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:00.388280 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.388186 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:38:00.388532 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.388367 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tsx8f\"" Oct 30 21:38:00.388532 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.388456 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:38:00.388532 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.388460 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:38:00.388532 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.388459 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:38:00.389463 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.389442 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8c95d677-txjhg"] Oct 30 21:38:00.391007 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.390956 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpg7q\" (UniqueName: \"kubernetes.io/projected/5230b5e1-f7df-4403-adbb-904ef729b5ab-kube-api-access-hpg7q\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:00.397726 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.397706 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:00.410610 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.410586 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41712999-d33f-467e-b3b8-dfa8c3fc650b-ca-trust-extracted\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.410720 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.410666 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zvr6\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-kube-api-access-8zvr6\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.410781 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.410735 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-trusted-ca\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.410831 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.410788 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a776136-2e97-427d-ad80-4b990b59d897-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qrpk7\" (UID: \"9a776136-2e97-427d-ad80-4b990b59d897\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:00.410831 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.410818 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-image-registry-private-configuration\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.410938 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.410843 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-certificates\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.410938 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.410889 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-bound-sa-token\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.411059 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.411040 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-tls\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.411112 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.411067 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-installation-pull-secrets\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.413251 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.413226 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a776136-2e97-427d-ad80-4b990b59d897-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qrpk7\" (UID: \"9a776136-2e97-427d-ad80-4b990b59d897\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:00.511715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.511683 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvr6\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-kube-api-access-8zvr6\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.511715 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.511718 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-trusted-ca\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.511929 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.511859 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-image-registry-private-configuration\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.511929 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.511892 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-certificates\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.511929 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.511908 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-bound-sa-token\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.512080 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.511943 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-tls\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.512131 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.512068 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-installation-pull-secrets\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.512131 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.512119 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41712999-d33f-467e-b3b8-dfa8c3fc650b-ca-trust-extracted\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.512502 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.512474 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41712999-d33f-467e-b3b8-dfa8c3fc650b-ca-trust-extracted\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.512724 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.512699 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-certificates\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.513110 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.513084 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-trusted-ca\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.514341 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.514316 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-tls\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.514476 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.514380 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-image-registry-private-configuration\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.514476 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.514466 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-installation-pull-secrets\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.519602 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.519444 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:00.583141 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.583116 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvr6\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-kube-api-access-8zvr6\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.586387 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.586360 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-bound-sa-token\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.821229 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:38:00.821201 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5230b5e1_f7df_4403_adbb_904ef729b5ab.slice/crio-17bf3a6302c964ed131626f75267d8d1057adcc9771f565715dde28628ad9749 WatchSource:0}: Error finding container 17bf3a6302c964ed131626f75267d8d1057adcc9771f565715dde28628ad9749: Status 404 returned error can't find the container with id 17bf3a6302c964ed131626f75267d8d1057adcc9771f565715dde28628ad9749 Oct 30 21:38:00.824931 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.822854 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-tvhwl"] Oct 30 21:38:00.881937 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.881909 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:00.885135 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:38:00.885108 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a776136_2e97_427d_ad80_4b990b59d897.slice/crio-473d99bbf46b769a53e9c79f144bcee0d9737c829fe228cab7cf740e599fff12 WatchSource:0}: Error finding container 473d99bbf46b769a53e9c79f144bcee0d9737c829fe228cab7cf740e599fff12: Status 404 returned error can't find the container with id 473d99bbf46b769a53e9c79f144bcee0d9737c829fe228cab7cf740e599fff12 Oct 30 21:38:00.888528 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:00.888505 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7"] Oct 30 21:38:01.149711 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.149620 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" event={"ID":"9a776136-2e97-427d-ad80-4b990b59d897","Type":"ContainerStarted","Data":"473d99bbf46b769a53e9c79f144bcee0d9737c829fe228cab7cf740e599fff12"} Oct 30 21:38:01.150599 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.150573 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tvhwl" event={"ID":"5230b5e1-f7df-4403-adbb-904ef729b5ab","Type":"ContainerStarted","Data":"17bf3a6302c964ed131626f75267d8d1057adcc9771f565715dde28628ad9749"} Oct 30 21:38:01.297385 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.297174 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6f8c95d677-txjhg"] Oct 30 21:38:01.298220 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:38:01.298179 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41712999_d33f_467e_b3b8_dfa8c3fc650b.slice/crio-41960a39f04f18fd7f5f7c6da3c23a039d54929485a7118adec91868afacd3b5 WatchSource:0}: Error finding container 41960a39f04f18fd7f5f7c6da3c23a039d54929485a7118adec91868afacd3b5: Status 404 returned error can't find the container with id 41960a39f04f18fd7f5f7c6da3c23a039d54929485a7118adec91868afacd3b5 Oct 30 21:38:01.886278 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.886244 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-55f66b6b69-xpmqp"] Oct 30 21:38:01.888182 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.888164 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:01.924398 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.924364 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:01.924551 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.924432 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:01.924551 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.924483 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:01.924551 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.924516 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd9p7\" (UniqueName: \"kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:01.924688 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.924619 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:01.924688 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.924645 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:01.924688 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.924663 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:01.949028 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.948990 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:38:01.949028 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.949027 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f66b6b69-xpmqp"] Oct 30 21:38:01.949210 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.949040 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:38:01.949210 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.949053 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-czbd6\"" Oct 30 21:38:01.949827 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.949808 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:38:01.949934 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.949911 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:38:01.949968 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:01.949960 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:38:02.025960 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.025911 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.026150 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.025986 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.026150 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.026011 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.026150 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.026033 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9p7\" (UniqueName: \"kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.026150 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.026081 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.026150 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.026106 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.026150 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.026126 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.026580 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.026556 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.027001 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.026977 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.027523 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.027499 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.027648 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.027629 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.029318 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.029296 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.029403 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.029334 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.090642 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.090607 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9p7\" (UniqueName: \"kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.156260 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.156165 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" event={"ID":"41712999-d33f-467e-b3b8-dfa8c3fc650b","Type":"ContainerStarted","Data":"e1945450e0e16f9914afc414403644f080b0ee447e16b37c1c345f419f7e96ad"} Oct 30 21:38:02.156260 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.156227 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" event={"ID":"41712999-d33f-467e-b3b8-dfa8c3fc650b","Type":"ContainerStarted","Data":"41960a39f04f18fd7f5f7c6da3c23a039d54929485a7118adec91868afacd3b5"} Oct 30 21:38:02.156789 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.156270 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:02.198230 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.198198 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:02.572394 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.571357 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" podStartSLOduration=2.571336611 podStartE2EDuration="2.571336611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:02.290629659 +0000 UTC m=+63.958112181" watchObservedRunningTime="2026-10-30 21:38:02.571336611 +0000 UTC m=+64.238819135" Oct 30 21:38:02.572394 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:38:02.571560 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251a9c03_36b1_463b_bf87_2d831dcd8419.slice/crio-4bad963001f4e985be372befd5f053f105d90af6c4af21cf67b44431c53306da WatchSource:0}: Error finding container 4bad963001f4e985be372befd5f053f105d90af6c4af21cf67b44431c53306da: Status 404 returned error can't find the container with id 4bad963001f4e985be372befd5f053f105d90af6c4af21cf67b44431c53306da Oct 30 21:38:02.572394 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:02.572325 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f66b6b69-xpmqp"] Oct 30 21:38:03.160275 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.160233 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f66b6b69-xpmqp" event={"ID":"251a9c03-36b1-463b-bf87-2d831dcd8419","Type":"ContainerStarted","Data":"4bad963001f4e985be372befd5f053f105d90af6c4af21cf67b44431c53306da"} Oct 30 21:38:03.162167 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.162132 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" event={"ID":"9a776136-2e97-427d-ad80-4b990b59d897","Type":"ContainerStarted","Data":"ea3743eecc4907284b29bda3a282a151d68ef26ffa91ac8080cd175909e35cef"} Oct 30 21:38:03.162523 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.162500 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:03.167964 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.167943 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:03.280074 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.280019 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" podStartSLOduration=1.436173181 podStartE2EDuration="3.280006755s" totalImagesPullingTime="1.843833574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:03.279254369 +0000 UTC m=+64.946736890" watchObservedRunningTime="2026-10-30 21:38:03.280006755 +0000 UTC m=+64.947489270" Oct 30 21:38:03.845660 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.845401 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:38:03.845660 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.845474 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:03.898179 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.898044 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:38:03.898179 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.898146 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:38:03.909261 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.909223 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:03.909770 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:03.909726 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:38:04.056629 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.056594 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7gkt\"" Oct 30 21:38:04.059541 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.059514 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v9xch\"" Oct 30 21:38:04.064750 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.064719 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:38:04.068132 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.068100 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:04.452903 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:38:04.452874 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod843e4020_8588_4ead_adda_ead6c9e0882b.slice/crio-17597a64cff7217396539308536c967a716869b7597d061a49ecdf1cc9e4a61e WatchSource:0}: Error finding container 17597a64cff7217396539308536c967a716869b7597d061a49ecdf1cc9e4a61e: Status 404 returned error can't find the container with id 17597a64cff7217396539308536c967a716869b7597d061a49ecdf1cc9e4a61e Oct 30 21:38:04.453766 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.453741 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-9rfmn"] Oct 30 21:38:04.499817 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.499792 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sbp5h"] Oct 30 21:38:04.499927 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:38:04.499798 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b80443a_852a_4e9e_8a6d_26de01cd2912.slice/crio-a41a23e4f241bd0971b668c491c556dc341a8a7f4cdc683d052698c2df05bae6 WatchSource:0}: Error finding container a41a23e4f241bd0971b668c491c556dc341a8a7f4cdc683d052698c2df05bae6: Status 404 returned error can't find the container with id a41a23e4f241bd0971b668c491c556dc341a8a7f4cdc683d052698c2df05bae6 Oct 30 21:38:04.583709 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.583672 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk"] Oct 30 21:38:04.587080 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.587051 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.643955 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.643908 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:38:04.651625 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.651320 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.651625 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.651365 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.651625 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.651407 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdq8q\" (UniqueName: \"kubernetes.io/projected/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-kube-api-access-mdq8q\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.651625 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.651475 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.694478 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.694369 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:38:04.694943 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.694814 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:38:04.694943 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.694854 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:38:04.694943 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.694877 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-28p6z\"" Oct 30 21:38:04.694943 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.694913 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:38:04.696025 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.696003 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk"] Oct 30 21:38:04.752759 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.752728 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.752968 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.752800 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.752968 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.752826 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.752968 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.752868 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdq8q\" (UniqueName: \"kubernetes.io/projected/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-kube-api-access-mdq8q\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.754643 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.754611 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.756122 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.756098 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.756351 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.756331 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.815975 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.815941 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdq8q\" (UniqueName: \"kubernetes.io/projected/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-kube-api-access-mdq8q\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:04.901171 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:04.901134 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:05.169049 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:05.168966 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rfmn" event={"ID":"843e4020-8588-4ead-adda-ead6c9e0882b","Type":"ContainerStarted","Data":"17597a64cff7217396539308536c967a716869b7597d061a49ecdf1cc9e4a61e"} Oct 30 21:38:05.170363 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:05.170330 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sbp5h" event={"ID":"9b80443a-852a-4e9e-8a6d-26de01cd2912","Type":"ContainerStarted","Data":"a41a23e4f241bd0971b668c491c556dc341a8a7f4cdc683d052698c2df05bae6"} Oct 30 21:38:05.278005 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:38:05.277978 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3f0931_4f7d_4849_9c3d_f09e25aadc18.slice/crio-6431ba4712370a2828871cb089d2d48ef4a28962412906af410f92d0b4756451 WatchSource:0}: Error finding container 6431ba4712370a2828871cb089d2d48ef4a28962412906af410f92d0b4756451: Status 404 returned error can't find the container with id 6431ba4712370a2828871cb089d2d48ef4a28962412906af410f92d0b4756451 Oct 30 21:38:05.278265 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:05.278240 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk"] Oct 30 21:38:05.559366 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:05.559327 2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:05.564867 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:05.564825 2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:05.859308 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:05.859239 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlcqp\"" Oct 30 21:38:05.867934 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:05.867903 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:05.889588 ip-10-0-133-5 kubenswrapper[2583]: E1030 21:38:05.888219 2583 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"serve-healthcheck-canary\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:bb08b4955abf8a0ffea0e4f2e446fbb6a5f3271312a996633901621b91b106ab\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage2702461699/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/2d/2d4d662444694496136a95671bc5ae4a2b7d8c9bbffc3f3a49c45608a64b4a27?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T213750Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=7926d18e553bd704b6b8164bab915c7df52cbd62373ea516ffc75877f5204ed2®ion=us-east-1&namespace=openshift-release-dev&username=openshift-release-dev+ocm_access_f62e7711b3d44085bacf45dd1a94949f&repo_name=ocp-v4.0-art-dev&akamai_signature=exp=1793397170~hmac=b0d648ba2205cf332e20c2e0d1e20fe78ff6b5541719a4f9624d9413f6bd748b\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="openshift-ingress-canary/ingress-canary-5gfv2" podUID="261c8437-5164-4544-bd94-90706f1d919b" Oct 30 21:38:06.175903 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:06.175774 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" event={"ID":"fe3f0931-4f7d-4849-9c3d-f09e25aadc18","Type":"ContainerStarted","Data":"6431ba4712370a2828871cb089d2d48ef4a28962412906af410f92d0b4756451"} Oct 30 21:38:06.269936 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:06.269909 2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fcsg4"] Oct 30 21:38:06.270460 ip-10-0-133-5 kubenswrapper[2583]: W1030 21:38:06.270411 2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42c37929_ba43_4002_877d_a3ab2f5723b8.slice/crio-7e3bbdc792c560f426412081fc938723d55b0216f8611664423e7ec12c53bc12 WatchSource:0}: Error finding container 7e3bbdc792c560f426412081fc938723d55b0216f8611664423e7ec12c53bc12: Status 404 returned error can't find the container with id 7e3bbdc792c560f426412081fc938723d55b0216f8611664423e7ec12c53bc12 Oct 30 21:38:07.085207 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:07.085175 2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:38:07.182578 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:07.182535 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sbp5h" event={"ID":"9b80443a-852a-4e9e-8a6d-26de01cd2912","Type":"ContainerStarted","Data":"bc55f0b2b27ae72b29f5a70b7c6924e545ac977efa165b39d005a1c9800fc363"} Oct 30 21:38:07.182757 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:07.182613 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sbp5h" event={"ID":"9b80443a-852a-4e9e-8a6d-26de01cd2912","Type":"ContainerStarted","Data":"7fe6845e3c0745eac26ae82eb874e190d69d9a879fd9d3d6a3d1ebc2e2cb72dd"} Oct 30 21:38:07.185298 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:07.185265 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f66b6b69-xpmqp" event={"ID":"251a9c03-36b1-463b-bf87-2d831dcd8419","Type":"ContainerStarted","Data":"7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9"} Oct 30 21:38:07.187766 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:07.187738 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fcsg4" event={"ID":"42c37929-ba43-4002-877d-a3ab2f5723b8","Type":"ContainerStarted","Data":"e052fe5eba7421440415962a8cae4bc2859a07bb8e18e755b7736143dd83a515"} Oct 30 21:38:07.187871 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:07.187800 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fcsg4" event={"ID":"42c37929-ba43-4002-877d-a3ab2f5723b8","Type":"ContainerStarted","Data":"7e3bbdc792c560f426412081fc938723d55b0216f8611664423e7ec12c53bc12"} Oct 30 21:38:07.441166 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:07.441057 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sbp5h" podStartSLOduration=65.550698103 podStartE2EDuration="1m7.44103887s" totalImagesPullingTime="1.890340767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:07.321633799 +0000 UTC m=+68.989116321" watchObservedRunningTime="2026-10-30 21:38:07.44103887 +0000 UTC m=+69.108521395" Oct 30 21:38:07.441330 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:07.441274 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-55f66b6b69-xpmqp" podStartSLOduration=2.674837647 podStartE2EDuration="6.44126453s" totalImagesPullingTime="3.766426883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:07.440636482 +0000 UTC m=+69.108119005" watchObservedRunningTime="2026-10-30 21:38:07.44126453 +0000 UTC m=+69.108747050" Oct 30 21:38:08.194629 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:08.194541 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" event={"ID":"fe3f0931-4f7d-4849-9c3d-f09e25aadc18","Type":"ContainerStarted","Data":"6c8f96f450229e6794dbb2e7ccf50d26678ccdb7723d1f1ea18704c33bdcfa06"} Oct 30 21:38:08.194629 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:08.194606 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" event={"ID":"fe3f0931-4f7d-4849-9c3d-f09e25aadc18","Type":"ContainerStarted","Data":"20ae5b802ecb8ee8b4eb2ef471653800ee2a1c1c76d7cf19e7dded43eeea0a23"} Oct 30 21:38:08.196799 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:08.196766 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fcsg4" event={"ID":"42c37929-ba43-4002-877d-a3ab2f5723b8","Type":"ContainerStarted","Data":"ba2a4b22eb0d5faa8829d66261dd8379490e9dbc2d4e0cc3a2d8c87646e739a9"} Oct 30 21:38:08.314351 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:08.313809 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" podStartSLOduration=1.8090232259999999 podStartE2EDuration="4.313789263s" totalImagesPullingTime="2.504766037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:08.312697536 +0000 UTC m=+69.980180089" watchObservedRunningTime="2026-10-30 21:38:08.313789263 +0000 UTC m=+69.981271788" Oct 30 21:38:11.215846 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.215801 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fcsg4" event={"ID":"42c37929-ba43-4002-877d-a3ab2f5723b8","Type":"ContainerStarted","Data":"6ebbf45df85a818876389008482004496297fa886945cb842d233b5376943601"} Oct 30 21:38:11.217983 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.217951 2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-9rfmn" event={"ID":"843e4020-8588-4ead-adda-ead6c9e0882b","Type":"ContainerStarted","Data":"886a0d507d0584a310964177e57f4a6611e91b8930af04954442321c318a7107"} Oct 30 21:38:11.229979 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.229948 2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-28r28"] Oct 30 21:38:11.232525 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.232504 2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.287514 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.287435 2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:38:11.287823 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.287796 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:38:11.287962 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.287855 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:38:11.287962 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.287892 2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zf5wc\"" Oct 30 21:38:11.307381 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307333 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-sys\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.307549 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307390 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-metrics-client-ca\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.307549 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307440 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlw2c\" (UniqueName: \"kubernetes.io/projected/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-kube-api-access-dlw2c\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.307659 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307601 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-textfile\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.307659 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307641 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-accelerators-collector-config\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.307739 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307665 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-tls\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.307739 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307683 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.307739 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307733 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-root\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.307897 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.307768 2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-wtmp\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:11.337956 ip-10-0-133-5 kubenswrapper[2583]: I1030 21:38:11.337889 2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-fcsg4" podStartSLOduration=34.616900527 podStartE2EDuration="38.337868856s" totalImagesPullingTime="3.720968329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:37:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:11.336491662 +0000 UTC m=+73.003974183" watchObservedRunningTime="2026-10-30 21:38:11.337868856 +0000 UTC m=+73.005351379" Oct 30 21:38:11.357557 ip-10-0-133-5 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 21:38:11.365084 ip-10-0-133-5 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 21:38:11.365298 ip-10-0-133-5 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 21:38:11.365369 ip-10-0-133-5 systemd[1]: kubelet.service: Consumed 3.549s CPU time. Oct 30 21:38:11.375149 ip-10-0-133-5 systemd[1]: Starting Kubernetes Kubelet... Oct 30 21:38:11.443266 ip-10-0-133-5 kubenswrapper[5139]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:38:11.443266 ip-10-0-133-5 kubenswrapper[5139]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 21:38:11.443266 ip-10-0-133-5 kubenswrapper[5139]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:38:11.443266 ip-10-0-133-5 kubenswrapper[5139]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 21:38:11.445844 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445748 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:11.445844 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445839 5139 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:11.445844 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445843 5139 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:11.445844 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445846 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:11.445844 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445849 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:11.445844 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445853 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:11.445844 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445856 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445860 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445863 5139 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445866 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445868 5139 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445872 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445874 5139 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445877 5139 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445880 5139 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445882 5139 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445885 5139 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445887 5139 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445890 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445892 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445895 5139 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445898 5139 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445908 5139 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445911 5139 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445914 5139 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445917 5139 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:11.446250 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445920 5139 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445923 5139 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445925 5139 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445928 5139 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445930 5139 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445933 5139 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445936 5139 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445938 5139 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445941 5139 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445944 5139 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445947 5139 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445950 5139 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445952 5139 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445955 5139 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445958 5139 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445961 5139 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445964 5139 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445966 5139 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445969 5139 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445971 5139 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:11.447141 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445975 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445977 5139 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445980 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445983 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445985 5139 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445989 5139 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445994 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445997 5139 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.445999 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446002 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446005 5139 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446007 5139 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446010 5139 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446013 5139 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446016 5139 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446019 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446022 5139 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446026 5139 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446029 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446031 5139 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:11.448013 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446034 5139 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446037 5139 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446039 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446042 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446044 5139 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446048 5139 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446051 5139 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446055 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446058 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446060 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446063 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446065 5139 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446068 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446071 5139 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446074 5139 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446076 5139 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446079 5139 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446082 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446084 5139 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:11.448917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446087 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446089 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446092 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446094 5139 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446097 5139 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446100 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446103 5139 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446106 5139 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446108 5139 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446111 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446113 5139 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446116 5139 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446118 5139 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446123 5139 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446127 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446130 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446133 5139 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446135 5139 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446138 5139 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:11.449748 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446141 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446144 5139 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446942 5139 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446950 5139 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446953 5139 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446957 5139 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446960 5139 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446962 5139 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446965 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446968 5139 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446970 5139 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446973 5139 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446976 5139 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446978 5139 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446981 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446986 5139 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446990 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446993 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446996 5139 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.446999 5139 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:11.450520 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447002 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447005 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447008 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447010 5139 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447012 5139 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447015 5139 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447018 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447021 5139 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447024 5139 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447027 5139 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447030 5139 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447033 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447035 5139 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447038 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447041 5139 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447044 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447047 5139 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447050 5139 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447052 5139 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447055 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:11.451367 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447057 5139 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447060 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447063 5139 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447065 5139 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447068 5139 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447070 5139 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447073 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447075 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447078 5139 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447081 5139 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447084 5139 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447086 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447089 5139 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447091 5139 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447094 5139 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447097 5139 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447099 5139 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447102 5139 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447104 5139 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447107 5139 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:11.452220 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447109 5139 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447112 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447114 5139 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447117 5139 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447120 5139 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447123 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447126 5139 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447130 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447132 5139 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447135 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447138 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447140 5139 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447143 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447146 5139 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447149 5139 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447152 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447155 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447159 5139 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447162 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:11.452883 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447165 5139 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447167 5139 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447170 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447173 5139 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447175 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447178 5139 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447181 5139 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447184 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447186 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447189 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447191 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447194 5139 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447197 5139 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447201 5139 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447205 5139 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447211 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447214 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447217 5139 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447220 5139 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:11.453378 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447222 5139 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447225 5139 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447228 5139 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447231 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447234 5139 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447236 5139 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447239 5139 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447242 5139 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447244 5139 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447247 5139 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447784 5139 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447791 5139 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447794 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447797 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447800 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447802 5139 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447805 5139 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447808 5139 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447811 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:11.453903 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447813 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447816 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447819 5139 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447822 5139 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447825 5139 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447828 5139 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447830 5139 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447833 5139 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447836 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447841 5139 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447844 5139 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447847 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447849 5139 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447852 5139 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447855 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447864 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447867 5139 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447869 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447872 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:11.454625 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447875 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447878 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447880 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447883 5139 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447886 5139 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447888 5139 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447891 5139 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447894 5139 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447898 5139 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447901 5139 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447905 5139 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447908 5139 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447911 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447913 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447916 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447919 5139 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447922 5139 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447924 5139 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447927 5139 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447929 5139 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:11.455395 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447932 5139 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447934 5139 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447938 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447941 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447944 5139 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447947 5139 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447949 5139 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447952 5139 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447955 5139 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447958 5139 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447961 5139 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447963 5139 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447966 5139 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447969 5139 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447972 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447974 5139 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447977 5139 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447980 5139 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447982 5139 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447985 5139 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:11.455944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447987 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447990 5139 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447993 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447995 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.447997 5139 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448000 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448003 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448005 5139 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448008 5139 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448010 5139 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448013 5139 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448017 5139 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448021 5139 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448024 5139 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448028 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448031 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448034 5139 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448037 5139 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448040 5139 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:11.456485 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448044 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448046 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448049 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448053 5139 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448056 5139 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448059 5139 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448061 5139 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448064 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448067 5139 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448069 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448072 5139 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448074 5139 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448077 5139 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448080 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448082 5139 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448085 5139 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448087 5139 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448090 5139 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448093 5139 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448188 5139 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448195 5139 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 21:38:11.456959 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448201 5139 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448206 5139 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448211 5139 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448214 5139 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448219 5139 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448223 5139 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448228 5139 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448231 5139 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448235 5139 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448238 5139 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448241 5139 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448246 5139 flags.go:64] FLAG: --cgroup-root="" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448249 5139 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448252 5139 flags.go:64] FLAG: --client-ca-file="" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448255 5139 flags.go:64] FLAG: --cloud-provider="external" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448259 5139 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448264 5139 flags.go:64] FLAG: --cluster-domain="" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448267 5139 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448270 5139 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448273 5139 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448277 5139 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448281 5139 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448284 5139 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 21:38:11.457561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448287 5139 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448291 5139 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448294 5139 flags.go:64] FLAG: --contention-profiling="false" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448298 5139 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448301 5139 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448304 5139 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448307 5139 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448312 5139 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448315 5139 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448318 5139 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448321 5139 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448324 5139 flags.go:64] FLAG: --enable-server="true" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448327 5139 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448331 5139 flags.go:64] FLAG: --event-burst="100" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448335 5139 flags.go:64] FLAG: --event-qps="50" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448339 5139 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448342 5139 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448345 5139 flags.go:64] FLAG: --eviction-hard="" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448349 5139 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448353 5139 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448358 5139 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448361 5139 flags.go:64] FLAG: --eviction-soft="" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448364 5139 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448367 5139 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448370 5139 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 21:38:11.458111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448375 5139 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448378 5139 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448381 5139 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448384 5139 flags.go:64] FLAG: --feature-gates="" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448394 5139 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448397 5139 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448401 5139 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448404 5139 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448407 5139 flags.go:64] FLAG: --healthz-port="10248" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448426 5139 flags.go:64] FLAG: --help="false" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448433 5139 flags.go:64] FLAG: --hostname-override="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448439 5139 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448444 5139 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448447 5139 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448451 5139 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448454 5139 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448457 5139 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448461 5139 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448464 5139 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448467 5139 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448470 5139 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448473 5139 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448477 5139 flags.go:64] FLAG: --kube-reserved="" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448481 5139 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 21:38:11.458747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448484 5139 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448487 5139 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448491 5139 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448495 5139 flags.go:64] FLAG: --lock-file="" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448498 5139 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448502 5139 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448505 5139 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448511 5139 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448514 5139 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448518 5139 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448521 5139 flags.go:64] FLAG: --logging-format="text" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448524 5139 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448528 5139 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448532 5139 flags.go:64] FLAG: --manifest-url="" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448535 5139 flags.go:64] FLAG: --manifest-url-header="" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448539 5139 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448543 5139 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448547 5139 flags.go:64] FLAG: --max-pods="110" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448550 5139 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448553 5139 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448557 5139 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448560 5139 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448563 5139 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448566 5139 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448569 5139 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 21:38:11.459328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448577 5139 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448580 5139 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448583 5139 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448587 5139 flags.go:64] FLAG: --pod-cidr="" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448590 5139 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448594 5139 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448597 5139 flags.go:64] FLAG: --pods-per-core="0" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448601 5139 flags.go:64] FLAG: --port="10250" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448604 5139 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448607 5139 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0600962a47c11e333" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448612 5139 flags.go:64] FLAG: --qos-reserved="" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448615 5139 flags.go:64] FLAG: --read-only-port="10255" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448618 5139 flags.go:64] FLAG: --register-node="true" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448622 5139 flags.go:64] FLAG: --register-with-taints="" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448626 5139 flags.go:64] FLAG: --registry-burst="10" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448629 5139 flags.go:64] FLAG: --registry-qps="5" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448632 5139 flags.go:64] FLAG: --reserved-cpus="" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448635 5139 flags.go:64] FLAG: --reserved-memory="" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448640 5139 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448643 5139 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448646 5139 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448649 5139 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448653 5139 flags.go:64] FLAG: --runonce="false" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448656 5139 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448659 5139 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448662 5139 flags.go:64] FLAG: --seccomp-default="false" Oct 30 21:38:11.459953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448665 5139 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448668 5139 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448672 5139 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448675 5139 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448678 5139 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448681 5139 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448684 5139 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448687 5139 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448691 5139 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448694 5139 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448697 5139 flags.go:64] FLAG: --system-cgroups="" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448702 5139 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448707 5139 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448711 5139 flags.go:64] FLAG: --tls-cert-file="" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448714 5139 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448720 5139 flags.go:64] FLAG: --tls-min-version="" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448725 5139 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448728 5139 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448731 5139 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448734 5139 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448738 5139 flags.go:64] FLAG: --v="2" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448742 5139 flags.go:64] FLAG: --version="false" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448746 5139 flags.go:64] FLAG: --vmodule="" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448750 5139 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.448755 5139 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 21:38:11.460605 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448900 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448904 5139 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448907 5139 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448910 5139 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448912 5139 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448915 5139 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448918 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448921 5139 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448924 5139 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448927 5139 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448930 5139 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448933 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448935 5139 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448938 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448941 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448944 5139 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448948 5139 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448952 5139 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448957 5139 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:11.461399 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448961 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448963 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448966 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448969 5139 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448973 5139 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448976 5139 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448979 5139 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448981 5139 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448984 5139 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448987 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448990 5139 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448992 5139 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448995 5139 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.448998 5139 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449001 5139 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449004 5139 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449006 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449009 5139 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449011 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449014 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:11.461933 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449017 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449019 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449022 5139 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449024 5139 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449027 5139 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449029 5139 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449032 5139 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449034 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449037 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449040 5139 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449042 5139 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449046 5139 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449049 5139 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449051 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449054 5139 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449056 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449060 5139 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449063 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449065 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449068 5139 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:11.462517 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449071 5139 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449073 5139 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449076 5139 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449078 5139 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449082 5139 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449085 5139 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449089 5139 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449092 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449094 5139 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449097 5139 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449100 5139 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449102 5139 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449105 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449107 5139 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449110 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449113 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449115 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449118 5139 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449121 5139 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:11.463065 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449123 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449126 5139 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449128 5139 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449131 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449135 5139 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449138 5139 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449141 5139 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449144 5139 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449146 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449151 5139 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449153 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449156 5139 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449159 5139 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449162 5139 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449164 5139 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449167 5139 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449169 5139 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449172 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449175 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:11.463587 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449177 5139 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449182 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449184 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449187 5139 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449190 5139 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449192 5139 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449194 5139 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449197 5139 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449199 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.449226 5139 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449263 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449268 5139 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449271 5139 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449273 5139 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449276 5139 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449279 5139 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:11.464047 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449281 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449285 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449288 5139 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449291 5139 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449294 5139 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449296 5139 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449299 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449303 5139 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449306 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449309 5139 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449311 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449314 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449316 5139 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449319 5139 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449322 5139 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449324 5139 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449327 5139 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449331 5139 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449339 5139 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449344 5139 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:11.464503 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449347 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449350 5139 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449353 5139 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449356 5139 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449358 5139 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449361 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449364 5139 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449366 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449369 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449372 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449375 5139 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449377 5139 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449380 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449385 5139 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449388 5139 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449390 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449393 5139 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449396 5139 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449399 5139 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449403 5139 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:11.465261 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449406 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449409 5139 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449425 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449430 5139 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449435 5139 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449438 5139 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449441 5139 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449443 5139 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449446 5139 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449449 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449452 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449456 5139 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449458 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449461 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449464 5139 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449466 5139 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449469 5139 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449472 5139 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449474 5139 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:11.466186 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449477 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449479 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449482 5139 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449485 5139 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449487 5139 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449490 5139 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449494 5139 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449497 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449500 5139 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449504 5139 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449507 5139 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449510 5139 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449514 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449517 5139 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449520 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449522 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449525 5139 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449530 5139 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449534 5139 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:11.466908 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449537 5139 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449541 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449544 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449547 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449550 5139 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449553 5139 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449556 5139 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449559 5139 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449561 5139 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449564 5139 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449566 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449569 5139 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449572 5139 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449574 5139 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449577 5139 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449579 5139 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449582 5139 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449585 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449587 5139 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449591 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:11.467369 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449594 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.449596 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.453470 5139 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.453488 5139 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.453507 5139 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.453513 5139 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453570 5139 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453575 5139 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453578 5139 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453581 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453584 5139 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453587 5139 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453589 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453592 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453595 5139 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453597 5139 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453600 5139 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453603 5139 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453605 5139 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:11.467940 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453608 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453611 5139 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453614 5139 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453616 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453619 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453622 5139 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453624 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453627 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453629 5139 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453632 5139 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453635 5139 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453637 5139 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453641 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453644 5139 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453646 5139 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453649 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453652 5139 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453657 5139 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453661 5139 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453665 5139 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:11.468504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453668 5139 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453671 5139 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453673 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453676 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453679 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453681 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453685 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453688 5139 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453691 5139 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453694 5139 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453696 5139 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453699 5139 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453702 5139 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453705 5139 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453707 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453710 5139 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453713 5139 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453715 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453718 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:11.469000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453721 5139 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453724 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453726 5139 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453729 5139 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453731 5139 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453734 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453737 5139 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453740 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453742 5139 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453745 5139 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453754 5139 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453758 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453760 5139 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453763 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453766 5139 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453769 5139 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453772 5139 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453774 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453777 5139 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453779 5139 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:11.469494 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453782 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453785 5139 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453787 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453790 5139 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453792 5139 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453795 5139 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453798 5139 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453801 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453806 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453809 5139 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453812 5139 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453814 5139 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453817 5139 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453820 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453823 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453825 5139 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453828 5139 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453831 5139 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453834 5139 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453836 5139 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:11.469985 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453839 5139 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453843 5139 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453847 5139 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453851 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453854 5139 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453857 5139 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453860 5139 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453862 5139 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453865 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453867 5139 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453870 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453873 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453876 5139 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.453878 5139 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.453908 5139 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455294 5139 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:11.470598 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455329 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455337 5139 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455359 5139 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455364 5139 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455369 5139 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455375 5139 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455380 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455385 5139 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455390 5139 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455397 5139 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455425 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455432 5139 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455509 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455649 5139 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455657 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455662 5139 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455666 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455671 5139 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:11.470994 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455676 5139 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455680 5139 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455684 5139 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455689 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455693 5139 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455697 5139 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455702 5139 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455706 5139 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455710 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455714 5139 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455718 5139 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455723 5139 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455727 5139 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455731 5139 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455735 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455747 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455752 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455756 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455760 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455765 5139 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:11.471439 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455769 5139 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455773 5139 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455778 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455782 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455786 5139 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455790 5139 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455794 5139 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455799 5139 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455803 5139 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455807 5139 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455811 5139 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455815 5139 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455820 5139 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455824 5139 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455830 5139 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455834 5139 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455838 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455843 5139 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455847 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455852 5139 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:11.471914 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455857 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455861 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455865 5139 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455870 5139 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455874 5139 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455879 5139 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455883 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455888 5139 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455893 5139 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455898 5139 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455904 5139 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455909 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455913 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455916 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455920 5139 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455923 5139 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455926 5139 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455928 5139 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455931 5139 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:11.472405 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455934 5139 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455936 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455939 5139 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455942 5139 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455945 5139 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455948 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455950 5139 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455953 5139 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455956 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455959 5139 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455962 5139 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455965 5139 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455967 5139 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455970 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455972 5139 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455975 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455978 5139 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455980 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455983 5139 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455986 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:11.472899 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455989 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455991 5139 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455994 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.455997 5139 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456000 5139 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456003 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456006 5139 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456008 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.456047 5139 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456087 5139 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456093 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456095 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456098 5139 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456101 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456103 5139 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456106 5139 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 21:38:11.473386 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456109 5139 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456111 5139 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456115 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456118 5139 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456121 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456125 5139 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456129 5139 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456132 5139 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456135 5139 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456138 5139 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456141 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456143 5139 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456146 5139 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456151 5139 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456156 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456159 5139 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456162 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456164 5139 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 21:38:11.473783 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456167 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456170 5139 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456173 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456175 5139 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456178 5139 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456180 5139 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456183 5139 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456186 5139 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456189 5139 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456191 5139 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456194 5139 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456197 5139 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456199 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456202 5139 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456205 5139 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456207 5139 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456210 5139 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456213 5139 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456216 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456218 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 21:38:11.474248 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456221 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456223 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456226 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456228 5139 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456231 5139 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456233 5139 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456236 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456238 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456241 5139 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456244 5139 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456246 5139 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456249 5139 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456252 5139 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456254 5139 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456257 5139 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456260 5139 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456262 5139 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456265 5139 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456268 5139 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456270 5139 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 21:38:11.474750 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456273 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456275 5139 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456278 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456280 5139 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456283 5139 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456286 5139 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456288 5139 feature_gate.go:386] unrecognized feature gate: Example Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456291 5139 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456294 5139 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456296 5139 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456299 5139 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456302 5139 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456304 5139 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456306 5139 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456309 5139 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456311 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456314 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456317 5139 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456319 5139 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 21:38:11.475236 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456322 5139 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456324 5139 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456326 5139 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456329 5139 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456332 5139 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456337 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456340 5139 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456342 5139 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456345 5139 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456348 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456350 5139 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456353 5139 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456355 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456358 5139 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456361 5139 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456364 5139 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456366 5139 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456369 5139 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456372 5139 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456374 5139 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 21:38:11.475718 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456378 5139 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:11.456381 5139 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.456463 5139 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.457478 5139 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.457556 5139 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.457994 5139 server.go:1014] "Starting client certificate rotation" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.458129 5139 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.458177 5139 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:31:59 +0000 UTC" deadline="2028-06-17 03:39:30.079269866 +0000 UTC" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.458203 5139 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14286h1m18.621068631s" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.458563 5139 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.461203 5139 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.462555 5139 log.go:25] "Validated CRI v1 runtime API" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.464350 5139 log.go:25] "Validated CRI v1 image API" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.467765 5139 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 21:38:11.476199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.469765 5139 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 80151f60-77cf-4281-9b03-0af3145fca58:/dev/nvme0n1p3 adaa7332-d728-43d7-93e6-c04291639958:/dev/nvme0n1p4] Oct 30 21:38:11.476578 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.469787 5139 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/039181eb63ea760ad52c55d4bf65f406ccd3f46ba3ed7e7183d0622396ab338c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/039181eb63ea760ad52c55d4bf65f406ccd3f46ba3ed7e7183d0622396ab338c/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14d1576a8ab84ef928bf6bb26d15c977b813ce17774d2237c745a34d8f499ddb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14d1576a8ab84ef928bf6bb26d15c977b813ce17774d2237c745a34d8f499ddb/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17597a64cff7217396539308536c967a716869b7597d061a49ecdf1cc9e4a61e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17597a64cff7217396539308536c967a716869b7597d061a49ecdf1cc9e4a61e/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17bf3a6302c964ed131626f75267d8d1057adcc9771f565715dde28628ad9749/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17bf3a6302c964ed131626f75267d8d1057adcc9771f565715dde28628ad9749/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/35ec724587d333a9d8f43a27f5c4330b96293de7b2a9f7870e5f6a90dfaa5a17/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/35ec724587d333a9d8f43a27f5c4330b96293de7b2a9f7870e5f6a90dfaa5a17/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37640b19f9282865b2bc93be6ab37f4497e2933dd4d16490d11ab717d26138a1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37640b19f9282865b2bc93be6ab37f4497e2933dd4d16490d11ab717d26138a1/userdata/shm major:0 minor:230 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3843cedccf2e1add68c02348e56020745d39f4044f652b4cf3350c4ea3ab6c5d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3843cedccf2e1add68c02348e56020745d39f4044f652b4cf3350c4ea3ab6c5d/userdata/shm major:0 minor:61 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3b07d769b1a4d468ce46a90757f89336ff03998c61ff7b572093f161c30fee89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3b07d769b1a4d468ce46a90757f89336ff03998c61ff7b572093f161c30fee89/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/41960a39f04f18fd7f5f7c6da3c23a039d54929485a7118adec91868afacd3b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/41960a39f04f18fd7f5f7c6da3c23a039d54929485a7118adec91868afacd3b5/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/473d99bbf46b769a53e9c79f144bcee0d9737c829fe228cab7cf740e599fff12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/473d99bbf46b769a53e9c79f144bcee0d9737c829fe228cab7cf740e599fff12/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/49ab811d4b3a6d0e71909de5e5344f69b2b09fee4400496ef0d5f4fec1e3bf8b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/49ab811d4b3a6d0e71909de5e5344f69b2b09fee4400496ef0d5f4fec1e3bf8b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4bad963001f4e985be372befd5f053f105d90af6c4af21cf67b44431c53306da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4bad963001f4e985be372befd5f053f105d90af6c4af21cf67b44431c53306da/userdata/shm major:0 minor:273 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52d00644bd4350dd3c1955975d721d6e21274bf684f44f3acc5e614bf27e4d5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52d00644bd4350dd3c1955975d721d6e21274bf684f44f3acc5e614bf27e4d5a/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5fb865af1615becd8b256e62b2c5e48e5844497a001664efd1fb5888170ddaeb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5fb865af1615becd8b256e62b2c5e48e5844497a001664efd1fb5888170ddaeb/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6431ba4712370a2828871cb089d2d48ef4a28962412906af410f92d0b4756451/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6431ba4712370a2828871cb089d2d48ef4a28962412906af410f92d0b4756451/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b19a7a890b1bf836805baa7d3ec840b7c58fbd334a7fa7bf067c20470e711f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b19a7a890b1bf836805baa7d3ec840b7c58fbd334a7fa7bf067c20470e711f3/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e3bbdc792c560f426412081fc938723d55b0216f8611664423e7ec12c53bc12/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e3bbdc792c560f426412081fc938723d55b0216f8611664423e7ec12c53bc12/userdata/shm major:0 minor:301 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a41a23e4f241bd0971b668c491c556dc341a8a7f4cdc683d052698c2df05bae6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a41a23e4f241bd0971b668c491c556dc341a8a7f4cdc683d052698c2df05bae6/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/abf664d805b9cc7669cee12830734f25b3feab54e406903a96c5461903ef7c4f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/abf664d805b9cc7669cee12830734f25b3feab54e406903a96c5461903ef7c4f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b75ca48e5a4ea8bdca39ae0aed91953075b66d381845b9ff8b2236b757e79364/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b75ca48e5a4ea8bdca39ae0aed91953075b66d381845b9ff8b2236b757e79364/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd3fed9922a28fad2449285780c43ced8f2188394a6fabe58ea5fab5f6777a05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd3fed9922a28fad2449285780c43ced8f2188394a6fabe58ea5fab5f6777a05/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cf97dafb596cc66eff58442e40d8f56d0468e9994a14f72fb0c3a30b1231b4a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cf97dafb596cc66eff58442e40d8f56d0468e9994a14f72fb0c3a30b1231b4a0/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/10088b05-b328-4162-8ef6-39e60f240fb3/volumes/kubernetes.io~projected/kube-api-access-wkg86:{mountpoint:/var/lib/kubelet/pods/10088b05-b328-4162-8ef6-39e60f240fb3/volumes/kubernetes.io~projected/kube-api-access-wkg86 major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/114097ab-0272-43e5-a492-f12db369977a/volumes/kubernetes.io~projected/kube-api-access-8nzqb:{mountpoint:/var/lib/kubelet/pods/114097ab-0272-43e5-a492-f12db369977a/volumes/kubernetes.io~projected/kube-api-access-8nzqb major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~projected/kube-api-access-sd9p7:{mountpoint:/var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~projected/kube-api-access-sd9p7 major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:271 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/261c8437-5164-4544-bd94-90706f1d919b/volumes/kubernetes.io~projected/kube-api-access-2sz7b:{mountpoint:/var/lib/kubelet/pods/261c8437-5164-4544-bd94-90706f1d919b/volumes/kubernetes.io~projected/kube-api-access-2sz7b major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/261c8437-5164-4544-bd94-90706f1d919b/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/261c8437-5164-4544-bd94-90706f1d919b/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:255 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/kube-api-access-8zvr6:{mountpoint:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/kube-api-access-8zvr6 major:0 minor:254 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/registry-tls:{mountpoint:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/registry-tls major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42c37929-ba43-4002-877d-a3ab2f5723b8/volumes/kubernetes.io~projected/kube-api-access-kr849:{mountpoint:/var/lib/kubelet/pods/42c37929-ba43-4002-877d-a3ab2f5723b8/volumes/kubernetes.io~projected/kube-api-access-kr849 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42c37929-ba43-4002-877d-a3ab2f5723b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/42c37929-ba43-4002-877d-a3ab2f5723b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:300 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5230b5e1-f7df-4403-adbb-904ef729b5ab/volumes/kubernetes.io~projected/kube-api-access-hpg7q:{mountpoint:/var/lib/kubelet/pods/5230b5e1-f7df-4403-adbb-904ef729b5ab/volumes/kubernetes.io~projected/kube-api-access-hpg7q major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/556b7537-074f-46ce-8db5-e1a0a7b94dca/volumes/kubernetes.io~projected/kube-api-access-88l6l:{mountpoint:/var/lib/kubelet/pods/556b7537-074f-46ce-8db5-e1a0a7b94dca/volumes/kubernetes.io~projected/kube-api-access-88l6l major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5aa67807-4df5-4e6f-a4f9-a378593abf94/volumes/kubernetes.io~projected/kube-api-access-2tqq2:{mountpoint:/var/lib/kubelet/pods/5aa67807-4df5-4e6f-a4f9-a378593abf94/volumes/kubernetes.io~projected/kube-api-access-2tqq2 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/708926c2-26a2-4888-ae37-a4c157a07e94/volumes/kubernetes.io~projected/kube-api-access-62lrs:{mountpoint:/var/lib/kubelet/pods/708926c2-26a2-4888-ae37-a4c157a07e94/volumes/kubernetes.io~projected/kube-api-access-62lrs major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/788ef145-30de-43f5-8249-799dbb536b1b/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/788ef145-30de-43f5-8249-799dbb536b1b/volumes/kubernetes.io~secret/agent-certs major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/843e4020-8588-4ead-adda-ead6c9e0882b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/843e4020-8588-4ead-adda-ead6c9e0882b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a49810a-da35-4032-b7ac-a6e3c7453d21/volumes/kubernetes.io~projected/kube-api-access-q6jqv:{mountpoint:/var/lib/kubelet/pods/9a49810a-da35-4032-b7ac-a6e3c7453d21/volumes/kubernetes.io~projected/kube-api-access-q6jqv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9a776136-2e97-427d-ad80-4b990b59d897/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/9a776136-2e97-427d-ad80-4b990b59d897/volumes/kubernetes.io~secret/tls-certificates major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b80443a-852a-4e9e-8a6d-26de01cd2912/volumes/kubernetes.io~projected/kube-api-access-kmt7z:{mountpoint:/var/lib/kubelet/pods/9b80443a-852a-4e9e-8a6d-26de01cd2912/volumes/kubernetes.io~projected/kube-api-access-kmt7z major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b80443a-852a-4e9e-8a6d-26de01cd2912/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/9b80443a-852a-4e9e-8a6d-26de01cd2912/volumes/kubernetes.io~secret/metrics-certs major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4d7df9d-6dbd-4b9f-a679-00d8d07db782/volumes/kubernetes.io~projected/kube-api-access-w78lc:{mountpoint:/var/lib/kubelet/pods/c4d7df9d-6dbd-4b9f-a679-00d8d07db782/volumes/kubernetes.io~projected/kube-api-access-w78lc major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~empty-dir/tmp major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~projected/kube-api-access-nstvb:{mountpoint:/var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~projected/kube-api-access-nstvb major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc/volumes/kubernetes.io~projected/kube-api-access-96mtw:{mountpoint:/var/lib/kubelet/pods/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc/volumes/kubernetes.io~projected/kube-api-access-96mtw major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volumes/kubernetes.io~projected/kube-api-access-nxkqn:{mountpoint:/var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volumes/kubernetes.io~projected/kube-api-access-nxkqn major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec88c90e-fce8-4f91-9d30-e554174ef776/volumes/kubernetes.io~projected/kube-api-access-mzrdr:{mountpoint:/var/lib/kubelet/pods/ec88c90e-fce8-4f91-9d30-e554174ef776/volumes/kubernetes.io~projected/kube-api-access-mzrdr major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ec88c90e-fce8-4f91-9d30-e554174ef776/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/ec88c90e-fce8-4f91-9d30-e554174ef776/volumes/kubernetes.io~secret/metrics-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~projected/kube-api-access-mdq8q:{mountpoint:/var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~projected/kube-api-access-mdq8q major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:294 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/896801c4d428ebdccb98437ea68bddc45c37e7ee29839778a842439632fe6e96/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ad36d35438a27136d11bde7bf3e15c3e66296e09d506229f79a5cdcb223bfc8b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/01d3d6fbf421e063af42becbc5d077cdab032fc3a506cb99edc8d8dba13ee203/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/875908b5d72e8c93b8a94005ffda952f5aaf886de0f0fb8bc9f3098eade4335b/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/97c5bc1de55965e2f60e01c5e087fec9076bfa08b6786c4d0cc9bcc9f2d187ad/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/23e92f5c8e5570a847b8e23b12d9e2b823e8189908711c58e30371cf5cd52214/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/45f7d0c32a790462b6141d26e5c5d7e0669268cf77a103c6a38f87dcfb43346d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/34257a9fcae13b57e4f74326c11ea081bd1e0a109424ee9a6a8aee4742876d78/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3fce1438c2cab6209c4520fac38a57e633fb1ad808151122397541eced8c951c/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b350f8a4e7b5149cc1d425ffd0384e01b4e691ca206e33e14ea47cfe126e9590/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/c7a75858169fed54c9d37a735ef71215a9637463f1e8462100e87c4bf50a55d7/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/a7980656488f59e5decd8146359e90ad4015baa630d734c882f4abd54888e9d6/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/f41d03e695cef2d6f1ce07c1316f75f8afd6df45ef27ae4bf9ffbad89ae93059/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cefab8b48bebfb667d737c7f1e04cb4baff219c23f466ff462da3e2b9f246fbf/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6bda6248a90e6c495c412dec8c4400d9935949e65763df9be3d7cb88718e9840/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f482527de72b20425300e825a9c9ec2e93c01763374873e251d97f6387a90207/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ed02ce224e464c659eb3644c5b668f5b883c99dcc05885923ab24b036cc7e759/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/5e4edf3b6cf1a7c55baf2b523fa1058b9f2f584270b1bdcc7fa85bed031f9971/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d0af4bac5a64ff85fa46584021206671358ca8260024af6bf0b9eb5cb0bd3553/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/5d0dd1e0edca622128e43df06fa3dc501bfa30e227316db40720b8ed7f22b940/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-189:{mountpoint:/var/lib/containers/storage/overlay/9d3f8fb01b5f17539bea2ec2a5db46b682a8121270bbe71d20b3b13e3a365159/merged major:0 minor:189 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/cf3faa158101adcb4c7945741eb95386fc107b70f47f3089c36924e4d023d098/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/ee17a45e3abc4ac4ed9acfa3e898f5b87b2175e64b04b6a3c1387e542507ece9/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/01e9f25c1f769352e5a201a416d23bae5268955a8d007ec3bcfbaba819345504/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/7f72099944bb028cb7c94a30a4293f82b88859fd086f2bfb46d4bfc3dfff873b/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/60ebc940da27669554e5bd0610f9991c69c545d322a764d66815140fc9a3dfaf/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/ab359ef569b39b65438189ef8383658d3fc7970dfbe87c18860b8c068800a330/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-234:{mountpoint:/var/lib/containers/storage/overlay/40b9c2117fdf276327ec30323d6d89633485ca8a7483d4005c4e4f685dda6aa8/merged major:0 minor:234 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/5c0a51550e77410d80bf4680d0eb90f6fdee8736fc7ec1d02902718c852ca1c2/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/628a308ed97c658a773c6b9462a7c756869697fd3ae408aab7ef07177125c9fa/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/7aaf77374e22012f58c022adcb6445d6f685d9dd41d7eb5b73dcec5fd723c4c8/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/24f5d6b8d454b905a5257aad9a72259536ea5ff29b9d9927eb1fc63764923812/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/0973b83150aa1f1bbe123e8da90a43734ee1d6164f7e9907361d43ae5f5ebb4f/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/17f9c71327891a466d4e7b540400882f22ebd41c8d1d246b987bf91b4948bb8c/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/9b3afc13ccfb2b599057c067ffc90bc60431d305ef73e1fb9ca795e7fe1ae847/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/a64e0d3286e9201069416fd6703b2a85df8a95c87b78f441ccdf97422858732f/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/925378ccd129800e5aeceed76404c17966650c2ccbcf39e84bc0eb1e515767d4/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/8b4b48cf978e48390c4148aa6436b435e90e481c774ca76db6bffe40b7ebdae2/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/b04b5bd9ea7583c7786e2f09327a1202093dd20f8a6ccc5c2e50719827fcb5ec/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/71d23831e87b34a0e7c367b9e39e381e9b9c1a9cc61f077baa33a46118f4620a/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/5ee6daef2e58d9bba355dcf232665dadaa86b4895a9db53614705fc6dfb30a4e/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/a4db3ed36e1d5170e2945669079cbd88e052e2febb08e6b998e5d58a6fa5d343/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/7181232748df6e39c5e6859c691449a0b7708b6ca20ba48dc52e2ed9cc9eb81d/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/09fc2b7c07f06eb049d2b6abb2f2e2750cde59ec32d4e9b90329564fa5dbfa9a/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-330:{mountpoint:/var/lib/containers/storage/overlay/6a8a000be7a7803fbfa6ec7c6fe57e4eae061599a0e2d619600390019a817abf/merged major:0 minor:330 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/5d19bd78493097518856ec0acd92baae07fd9f5711349ff5cbac07509ac95000/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/0d55070fbf6603101aa5ac8935caf77efa4a1d0ac7d283bcf9c5d0d0955dc6a7/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/6d28b34061ce11e2f22526a81b3c1380fdd2987fff6157955e8c7f7d8c869eb1/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-64:{mountpoint:/var/lib/containers/storage/overlay/d0463a8eb15a865d6de36c8fc5214e11f8bfeec2c2de9a5c09065caab4a0423d/merged major:0 minor:64 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/56623a0779643cff6c53c919f90909352fc7c85a525cc4628aea832df4a06ef1/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/9f03c716f4d26fdabb0906a4f3088ca4b1d15eaa789fa996d314a057f2173217/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/75fbdb96a8f50c45a179739b5eafbd5faec9ede051339605b8c864992f979e84/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/eb6dcb9facf5503d1b224ecba3de7919b4ff76832c2c078b94db77c5ead9cd04/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/783c5c627bc5aa47e1f35f5539b739af9be3943ae2a839ad53180c438f5ef0d7/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/f94abbdd6e38838c453ea6efd538bf898c7ac89bfc5031b50ed918500b78c4ac/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/66cb1c14a426dba533e491979d6a76980fffd7ac4e618588e558441a2c05462f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/189068c5cd60acbd240172821e8a0d6ccb8e8771355ec5c5156f21ab16650c67/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/0ce3e71dc3665a2df510bd917f3078b31c4dbd7730c7aa326317f86fa275abf7/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 21:38:11.477022 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.476692 5139 manager.go:217] Machine: {Timestamp:2026-10-30 21:38:11.476204631 +0000 UTC m=+0.067490109 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3108474 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23cfecc3c3ef299507dfb7c067a6d7 SystemUUID:ec23cfec-c3c3-ef29-9507-dfb7c067a6d7 BootID:b02f8f5d-675b-48fd-9dd8-fa171e4352b6 Filesystems:[{Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:255 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/788ef145-30de-43f5-8249-799dbb536b1b/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:48 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c4d7df9d-6dbd-4b9f-a679-00d8d07db782/volumes/kubernetes.io~projected/kube-api-access-w78lc DeviceMajor:0 DeviceMinor:51 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/10088b05-b328-4162-8ef6-39e60f240fb3/volumes/kubernetes.io~projected/kube-api-access-wkg86 DeviceMajor:0 DeviceMinor:53 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-189 DeviceMajor:0 DeviceMinor:189 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/42c37929-ba43-4002-877d-a3ab2f5723b8/volumes/kubernetes.io~projected/kube-api-access-kr849 DeviceMajor:0 DeviceMinor:211 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6431ba4712370a2828871cb089d2d48ef4a28962412906af410f92d0b4756451/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/261c8437-5164-4544-bd94-90706f1d919b/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:49 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/039181eb63ea760ad52c55d4bf65f406ccd3f46ba3ed7e7183d0622396ab338c/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3843cedccf2e1add68c02348e56020745d39f4044f652b4cf3350c4ea3ab6c5d/userdata/shm DeviceMajor:0 DeviceMinor:61 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14d1576a8ab84ef928bf6bb26d15c977b813ce17774d2237c745a34d8f499ddb/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17bf3a6302c964ed131626f75267d8d1057adcc9771f565715dde28628ad9749/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3b07d769b1a4d468ce46a90757f89336ff03998c61ff7b572093f161c30fee89/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e3bbdc792c560f426412081fc938723d55b0216f8611664423e7ec12c53bc12/userdata/shm DeviceMajor:0 DeviceMinor:301 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/473d99bbf46b769a53e9c79f144bcee0d9737c829fe228cab7cf740e599fff12/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/35ec724587d333a9d8f43a27f5c4330b96293de7b2a9f7870e5f6a90dfaa5a17/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-234 DeviceMajor:0 DeviceMinor:234 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:250 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/42c37929-ba43-4002-877d-a3ab2f5723b8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:300 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:50 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:293 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d757110b-4b02-4857-853b-949b53441428/volumes/kubernetes.io~projected/kube-api-access-nstvb DeviceMajor:0 DeviceMinor:52 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5fb865af1615becd8b256e62b2c5e48e5844497a001664efd1fb5888170ddaeb/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:271 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9b80443a-852a-4e9e-8a6d-26de01cd2912/volumes/kubernetes.io~projected/kube-api-access-kmt7z DeviceMajor:0 DeviceMinor:54 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37640b19f9282865b2bc93be6ab37f4497e2933dd4d16490d11ab717d26138a1/userdata/shm DeviceMajor:0 DeviceMinor:230 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9b80443a-852a-4e9e-8a6d-26de01cd2912/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:284 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/abf664d805b9cc7669cee12830734f25b3feab54e406903a96c5461903ef7c4f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:47 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/eab1dc97-e877-4b0f-ab0c-406eb137b296/volumes/kubernetes.io~projected/kube-api-access-nxkqn DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b75ca48e5a4ea8bdca39ae0aed91953075b66d381845b9ff8b2236b757e79364/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec88c90e-fce8-4f91-9d30-e554174ef776/volumes/kubernetes.io~projected/kube-api-access-mzrdr DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/registry-tls DeviceMajor:0 DeviceMinor:249 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:251 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd3fed9922a28fad2449285780c43ced8f2188394a6fabe58ea5fab5f6777a05/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/261c8437-5164-4544-bd94-90706f1d919b/volumes/kubernetes.io~projected/kube-api-access-2sz7b DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a41a23e4f241bd0971b668c491c556dc341a8a7f4cdc683d052698c2df05bae6/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-330 DeviceMajor:0 DeviceMinor:330 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9a49810a-da35-4032-b7ac-a6e3c7453d21/volumes/kubernetes.io~projected/kube-api-access-q6jqv DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~projected/kube-api-access-mdq8q DeviceMajor:0 DeviceMinor:295 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/556b7537-074f-46ce-8db5-e1a0a7b94dca/volumes/kubernetes.io~projected/kube-api-access-88l6l DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9a776136-2e97-427d-ad80-4b990b59d897/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:248 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:266 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4bad963001f4e985be372befd5f053f105d90af6c4af21cf67b44431c53306da/userdata/shm DeviceMajor:0 DeviceMinor:273 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b19a7a890b1bf836805baa7d3ec840b7c58fbd334a7fa7bf067c20470e711f3/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/843e4020-8588-4ead-adda-ead6c9e0882b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:279 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/fe3f0931-4f7d-4849-9c3d-f09e25aadc18/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:294 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-64 DeviceMajor:0 DeviceMinor:64 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc/volumes/kubernetes.io~projected/kube-api-access-96mtw DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/49ab811d4b3a6d0e71909de5e5344f69b2b09fee4400496ef0d5f4fec1e3bf8b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/708926c2-26a2-4888-ae37-a4c157a07e94/volumes/kubernetes.io~projected/kube-api-access-62lrs DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ec88c90e-fce8-4f91-9d30-e554174ef776/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:225 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes/kubernetes.io~projected/kube-api-access-sd9p7 DeviceMajor:0 DeviceMinor:272 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17597a64cff7217396539308536c967a716869b7597d061a49ecdf1cc9e4a61e/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52d00644bd4350dd3c1955975d721d6e21274bf684f44f3acc5e614bf27e4d5a/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/41712999-d33f-467e-b3b8-dfa8c3fc650b/volumes/kubernetes.io~projected/kube-api-access-8zvr6 DeviceMajor:0 DeviceMinor:254 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cf97dafb596cc66eff58442e40d8f56d0468e9994a14f72fb0c3a30b1231b4a0/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/114097ab-0272-43e5-a492-f12db369977a/volumes/kubernetes.io~projected/kube-api-access-8nzqb DeviceMajor:0 DeviceMinor:212 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5230b5e1-f7df-4403-adbb-904ef729b5ab/volumes/kubernetes.io~projected/kube-api-access-hpg7q DeviceMajor:0 DeviceMinor:242 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/41960a39f04f18fd7f5f7c6da3c23a039d54929485a7118adec91868afacd3b5/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5aa67807-4df5-4e6f-a4f9-a378593abf94/volumes/kubernetes.io~projected/kube-api-access-2tqq2 DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:14d1576a8ab84ef MacAddress:5e:80:e8:db:19:34 Speed:10000 Mtu:8901} {Name:17597a64cff7217 MacAddress:0a:34:3d:41:bf:1f Speed:10000 Mtu:8901} {Name:17bf3a6302c964e MacAddress:92:81:96:8f:fd:46 Speed:10000 Mtu:8901} {Name:37640b19f928286 MacAddress:6a:09:3e:8e:22:58 Speed:10000 Mtu:8901} {Name:3b07d769b1a4d46 MacAddress:62:e8:e8:92:12:72 Speed:10000 Mtu:8901} {Name:41960a39f04f18f MacAddress:ce:02:59:1a:b2:11 Speed:10000 Mtu:8901} {Name:473d99bbf46b769 MacAddress:3a:44:34:bb:a1:cc Speed:10000 Mtu:8901} {Name:4bad963001f4e98 MacAddress:be:b7:9b:7c:ec:5c Speed:10000 Mtu:8901} {Name:6431ba4712370a2 MacAddress:82:96:5d:79:16:0e Speed:10000 Mtu:8901} {Name:7b19a7a890b1bf8 MacAddress:ae:9d:0f:49:a9:fd Speed:10000 Mtu:8901} {Name:7e3bbdc792c560f MacAddress:06:c6:43:bf:ae:04 Speed:10000 Mtu:8901} {Name:a41a23e4f241bd0 MacAddress:6a:65:94:4e:02:e4 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f9:48:58:33 Speed:0 Mtu:9001} {Name:br-int MacAddress:2e:6a:78:6b:72:ff Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:f9:48:58:33 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:ae:3d:78:57:48:c7 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:da:83:22:70:99:10 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 21:38:11.477185 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.477026 5139 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 21:38:11.477185 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.477077 5139 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 21:38:11.477480 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.477458 5139 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 21:38:11.477853 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.477481 5139 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-133-5.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 21:38:11.477945 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.477859 5139 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 21:38:11.477945 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.477874 5139 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 21:38:11.477945 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.477895 5139 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:38:11.477945 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.477922 5139 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 21:38:11.478113 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.478057 5139 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 21:38:11.478910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.478360 5139 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 21:38:11.478910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.478535 5139 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 21:38:11.478910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.478578 5139 kubelet.go:493] "Attempting to sync node with API server" Oct 30 21:38:11.478910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.478599 5139 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 21:38:11.478910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.478616 5139 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 21:38:11.478910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.478629 5139 kubelet.go:405] "Adding apiserver pod source" Oct 30 21:38:11.478910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.478642 5139 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 21:38:11.480520 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.480499 5139 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:38:11.480520 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.480519 5139 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:38:11.481111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.481092 5139 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 21:38:11.493126 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493102 5139 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 21:38:11.493126 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493119 5139 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 21:38:11.493289 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493236 5139 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 21:38:11.493480 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493465 5139 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 21:38:11.493515 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493498 5139 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 21:38:11.493616 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493606 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 21:38:11.493662 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493622 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 21:38:11.493662 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493630 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 21:38:11.493662 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493636 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 21:38:11.493662 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493646 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 21:38:11.493662 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493661 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 21:38:11.493924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493673 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 21:38:11.493924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493681 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 21:38:11.493924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493697 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 21:38:11.493924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493717 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 21:38:11.493924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493745 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 21:38:11.493924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493758 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 21:38:11.493924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493789 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 21:38:11.493924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.493794 5139 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 21:38:11.497536 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.497519 5139 server.go:1263] "Started kubelet" Oct 30 21:38:11.497673 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.497622 5139 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 21:38:11.497728 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.497692 5139 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 21:38:11.497728 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.497656 5139 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 21:38:11.498182 ip-10-0-133-5 systemd[1]: Started Kubernetes Kubelet. Oct 30 21:38:11.498330 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.498307 5139 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 21:38:11.499657 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.499574 5139 server.go:317] "Adding debug handlers to kubelet server" Oct 30 21:38:11.503210 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.503191 5139 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 21:38:11.503727 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.503709 5139 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 21:38:11.503812 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.503767 5139 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 21:38:11.503812 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.503773 5139 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 21:38:11.503913 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.503817 5139 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 21:38:11.504131 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.504106 5139 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" Oct 30 21:38:11.504993 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.504482 5139 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:32:00 +0000 UTC" deadline="2028-04-05 21:43:21.957603025 +0000 UTC" Oct 30 21:38:11.504993 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.504527 5139 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12552h5m10.453080108s" Oct 30 21:38:11.504993 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.504977 5139 factory.go:55] Registering systemd factory Oct 30 21:38:11.504993 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.504995 5139 factory.go:223] Registration of the systemd container factory successfully Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.506715 5139 factory.go:153] Registering CRI-O factory Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.506739 5139 factory.go:223] Registration of the crio container factory successfully Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.506813 5139 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.506839 5139 factory.go:103] Registering Raw factory Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.506858 5139 manager.go:1199] Started watching for new ooms in manager Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.507284 5139 manager.go:319] Starting recovery of all containers Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508116 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d757110b-4b02-4857-853b-949b53441428" volumeName="kubernetes.io/projected/d757110b-4b02-4857-853b-949b53441428-kube-api-access-nstvb" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508162 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab1dc97-e877-4b0f-ab0c-406eb137b296" volumeName="kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-config" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508186 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41712999-d33f-467e-b3b8-dfa8c3fc650b" volumeName="kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-installation-pull-secrets" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508201 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42c37929-ba43-4002-877d-a3ab2f5723b8" volumeName="kubernetes.io/empty-dir/42c37929-ba43-4002-877d-a3ab2f5723b8-data-volume" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508211 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42c37929-ba43-4002-877d-a3ab2f5723b8" volumeName="kubernetes.io/projected/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-api-access-kr849" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508219 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251a9c03-36b1-463b-bf87-2d831dcd8419" volumeName="kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508227 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41712999-d33f-467e-b3b8-dfa8c3fc650b" volumeName="kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-tls" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508236 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="843e4020-8588-4ead-adda-ead6c9e0882b" volumeName="kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508266 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4d7df9d-6dbd-4b9f-a679-00d8d07db782" volumeName="kubernetes.io/projected/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-kube-api-access-w78lc" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508274 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3f0931-4f7d-4849-9c3d-f09e25aadc18" volumeName="kubernetes.io/configmap/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-metrics-client-ca" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508295 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251a9c03-36b1-463b-bf87-2d831dcd8419" volumeName="kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508304 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3f0931-4f7d-4849-9c3d-f09e25aadc18" volumeName="kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508314 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42c37929-ba43-4002-877d-a3ab2f5723b8" volumeName="kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508321 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aa67807-4df5-4e6f-a4f9-a378593abf94" volumeName="kubernetes.io/empty-dir/5aa67807-4df5-4e6f-a4f9-a378593abf94-tmp-dir" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508331 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab1dc97-e877-4b0f-ab0c-406eb137b296" volumeName="kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-env-overrides" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508344 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="261c8437-5164-4544-bd94-90706f1d919b" volumeName="kubernetes.io/projected/261c8437-5164-4544-bd94-90706f1d919b-kube-api-access-2sz7b" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508369 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="708926c2-26a2-4888-ae37-a4c157a07e94" volumeName="kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508377 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc" volumeName="kubernetes.io/projected/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-kube-api-access-96mtw" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508386 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251a9c03-36b1-463b-bf87-2d831dcd8419" volumeName="kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508396 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251a9c03-36b1-463b-bf87-2d831dcd8419" volumeName="kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508406 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d757110b-4b02-4857-853b-949b53441428" volumeName="kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-tmp" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508429 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41712999-d33f-467e-b3b8-dfa8c3fc650b" volumeName="kubernetes.io/empty-dir/41712999-d33f-467e-b3b8-dfa8c3fc650b-ca-trust-extracted" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508463 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41712999-d33f-467e-b3b8-dfa8c3fc650b" volumeName="kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-image-registry-private-configuration" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508474 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4d7df9d-6dbd-4b9f-a679-00d8d07db782" volumeName="kubernetes.io/configmap/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-iptables-alerter-script" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508483 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10088b05-b328-4162-8ef6-39e60f240fb3" volumeName="kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508504 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a776136-2e97-427d-ad80-4b990b59d897" volumeName="kubernetes.io/secret/9a776136-2e97-427d-ad80-4b990b59d897-tls-certificates" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508513 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251a9c03-36b1-463b-bf87-2d831dcd8419" volumeName="kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508523 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab1dc97-e877-4b0f-ab0c-406eb137b296" volumeName="kubernetes.io/secret/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508536 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec88c90e-fce8-4f91-9d30-e554174ef776" volumeName="kubernetes.io/projected/ec88c90e-fce8-4f91-9d30-e554174ef776-kube-api-access-mzrdr" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508546 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251a9c03-36b1-463b-bf87-2d831dcd8419" volumeName="kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508568 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41712999-d33f-467e-b3b8-dfa8c3fc650b" volumeName="kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-certificates" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508581 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="788ef145-30de-43f5-8249-799dbb536b1b" volumeName="kubernetes.io/configmap/788ef145-30de-43f5-8249-799dbb536b1b-konnectivity-ca" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508590 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b80443a-852a-4e9e-8a6d-26de01cd2912" volumeName="kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508602 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab1dc97-e877-4b0f-ab0c-406eb137b296" volumeName="kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-script-lib" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508613 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec88c90e-fce8-4f91-9d30-e554174ef776" volumeName="kubernetes.io/empty-dir/ec88c90e-fce8-4f91-9d30-e554174ef776-tmp-dir" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508622 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42c37929-ba43-4002-877d-a3ab2f5723b8" volumeName="kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508631 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41712999-d33f-467e-b3b8-dfa8c3fc650b" volumeName="kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-bound-sa-token" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508639 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d757110b-4b02-4857-853b-949b53441428" volumeName="kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-etc-tuned" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508648 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3f0931-4f7d-4849-9c3d-f09e25aadc18" volumeName="kubernetes.io/projected/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-kube-api-access-mdq8q" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508658 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="114097ab-0272-43e5-a492-f12db369977a" volumeName="kubernetes.io/projected/114097ab-0272-43e5-a492-f12db369977a-kube-api-access-8nzqb" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508667 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5230b5e1-f7df-4403-adbb-904ef729b5ab" volumeName="kubernetes.io/empty-dir/5230b5e1-f7df-4403-adbb-904ef729b5ab-tmp" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508676 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5230b5e1-f7df-4403-adbb-904ef729b5ab" volumeName="kubernetes.io/projected/5230b5e1-f7df-4403-adbb-904ef729b5ab-kube-api-access-hpg7q" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508685 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="788ef145-30de-43f5-8249-799dbb536b1b" volumeName="kubernetes.io/secret/788ef145-30de-43f5-8249-799dbb536b1b-agent-certs" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508693 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe3f0931-4f7d-4849-9c3d-f09e25aadc18" volumeName="kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-tls" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508740 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5aa67807-4df5-4e6f-a4f9-a378593abf94" volumeName="kubernetes.io/projected/5aa67807-4df5-4e6f-a4f9-a378593abf94-kube-api-access-2tqq2" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508757 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a49810a-da35-4032-b7ac-a6e3c7453d21" volumeName="kubernetes.io/projected/9a49810a-da35-4032-b7ac-a6e3c7453d21-kube-api-access-q6jqv" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508773 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eab1dc97-e877-4b0f-ab0c-406eb137b296" volumeName="kubernetes.io/projected/eab1dc97-e877-4b0f-ab0c-406eb137b296-kube-api-access-nxkqn" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508786 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec88c90e-fce8-4f91-9d30-e554174ef776" volumeName="kubernetes.io/configmap/ec88c90e-fce8-4f91-9d30-e554174ef776-config-volume" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508800 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10088b05-b328-4162-8ef6-39e60f240fb3" volumeName="kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508807 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10088b05-b328-4162-8ef6-39e60f240fb3" volumeName="kubernetes.io/projected/10088b05-b328-4162-8ef6-39e60f240fb3-kube-api-access-wkg86" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508817 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="251a9c03-36b1-463b-bf87-2d831dcd8419" volumeName="kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508828 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41712999-d33f-467e-b3b8-dfa8c3fc650b" volumeName="kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-trusted-ca" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508871 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="556b7537-074f-46ce-8db5-e1a0a7b94dca" volumeName="kubernetes.io/projected/556b7537-074f-46ce-8db5-e1a0a7b94dca-kube-api-access-88l6l" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508886 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b80443a-852a-4e9e-8a6d-26de01cd2912" volumeName="kubernetes.io/projected/9b80443a-852a-4e9e-8a6d-26de01cd2912-kube-api-access-kmt7z" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508905 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="10088b05-b328-4162-8ef6-39e60f240fb3" volumeName="kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-binary-copy" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508941 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41712999-d33f-467e-b3b8-dfa8c3fc650b" volumeName="kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-kube-api-access-8zvr6" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508964 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9a49810a-da35-4032-b7ac-a6e3c7453d21" volumeName="kubernetes.io/configmap/9a49810a-da35-4032-b7ac-a6e3c7453d21-serviceca" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508975 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc" volumeName="kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cni-binary-copy" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508984 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ec88c90e-fce8-4f91-9d30-e554174ef776" volumeName="kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.508993 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="261c8437-5164-4544-bd94-90706f1d919b" volumeName="kubernetes.io/secret/261c8437-5164-4544-bd94-90706f1d919b-cert" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.509001 5139 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc" volumeName="kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-daemon-config" seLinuxMountContext="" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.509011 5139 reconstruct.go:97] "Volume reconstruction finished" Oct 30 21:38:11.509350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.509018 5139 reconciler.go:29] "Reconciler: start to sync state" Oct 30 21:38:11.533293 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.533216 5139 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 21:38:11.534941 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.534909 5139 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 21:38:11.534941 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.534934 5139 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 21:38:11.535113 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.534956 5139 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 21:38:11.535113 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.534997 5139 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 21:38:11.559267 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.559191 5139 generic.go:356] "Generic (PLEG): container finished" podID="1d8f9506264df06e876fee4a0052af02" containerID="f5217bdad722aafd4341c65a44fc35bf2ed9e33e5e0623f6dce6afcc45e91c62" exitCode=0 Oct 30 21:38:11.566186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.566154 5139 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="1eb9e3ccd035eb9115e69d68551f7878c8bd8c4e19db533dddc88c5c781dd2a8" exitCode=0 Oct 30 21:38:11.566186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.566180 5139 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="fc108a82d389a5c0587529204efc6e21b790206197f2d19a10b684cc0abca5f1" exitCode=0 Oct 30 21:38:11.566186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.566186 5139 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="82b6da202d537685cffde05ec06d7eb4c299c95de50d8056724c4189938e57fe" exitCode=0 Oct 30 21:38:11.566186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.566193 5139 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="bc32484bfe7c64fc4b8cd34caa323e1edec9e7a98a8b5eefcf7c894feafcf956" exitCode=0 Oct 30 21:38:11.566428 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.566200 5139 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="00f5ada52b43c2ca91fdbbde9f1a62508ec01d5a8d79b0db9737a3a4f785c390" exitCode=0 Oct 30 21:38:11.566428 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.566206 5139 generic.go:356] "Generic (PLEG): container finished" podID="10088b05-b328-4162-8ef6-39e60f240fb3" containerID="b675ba0afc64c8dd9e5d82ecb871fe51cd84744c341f0760f3884bd1d7c3fe41" exitCode=0 Oct 30 21:38:11.598049 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.598028 5139 manager.go:324] Recovery completed Oct 30 21:38:11.604180 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.604159 5139 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" Oct 30 21:38:11.614376 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.614359 5139 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:11.616329 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.616312 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:11.616434 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.616342 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:11.616434 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.616358 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:11.619012 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.618994 5139 cpu_manager.go:226] "Starting" policy="none" Oct 30 21:38:11.619067 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619013 5139 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 21:38:11.619067 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619030 5139 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 21:38:11.619121 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619004 5139 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:11.619178 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619163 5139 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 21:38:11.619212 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619178 5139 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 21:38:11.619212 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619197 5139 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 21:38:11.619212 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619202 5139 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 21:38:11.619212 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619207 5139 policy_none.go:50] "Start" Oct 30 21:38:11.619340 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619216 5139 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 21:38:11.619340 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619226 5139 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 21:38:11.619438 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619399 5139 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 21:38:11.619438 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619408 5139 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 21:38:11.619438 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.619432 5139 policy_none.go:44] "Start" Oct 30 21:38:11.622388 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.620238 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:11.622388 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.620287 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:11.622388 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.620303 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:11.625328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.625298 5139 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 21:38:11.630583 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.630569 5139 manager.go:341] "Starting Device Plugin manager" Oct 30 21:38:11.630685 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.630599 5139 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 21:38:11.630685 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.630609 5139 server.go:87] "Starting device plugin registration server" Oct 30 21:38:11.630895 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.630882 5139 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 21:38:11.630954 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.630893 5139 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 21:38:11.631011 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.630990 5139 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 21:38:11.631145 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.631103 5139 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 21:38:11.631262 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.631155 5139 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 21:38:11.631262 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.631162 5139 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 21:38:11.631382 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.631220 5139 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:38:11.631137723Z","UUID":"a44a8145-c4f7-46b6-9dca-64458713c3f7","Handler":null,"Name":"","Endpoint":""} Oct 30 21:38:11.632706 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.632670 5139 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 21:38:11.632810 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.632714 5139 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 21:38:11.632810 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.632787 5139 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 21:38:11.632924 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.632818 5139 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" Oct 30 21:38:11.635223 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.635187 5139 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal"] Oct 30 21:38:11.635341 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.635326 5139 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:11.636251 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.636231 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:11.636336 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.636262 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:11.636336 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.636281 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:11.636597 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.636583 5139 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:11.636750 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.636734 5139 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:11.637392 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637373 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:11.637392 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637382 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:11.637548 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637400 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:11.637548 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637405 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:11.637548 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637431 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:11.637548 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637432 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:11.637707 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.637591 5139 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.637780 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637751 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"3ab1a8fb7b394ed5c2ad3173d5294ab9","Type":"ContainerStarted","Data":"2eec01a4e462ad04d77e4fbdc1e9ede6d5b8ab8d413f8ecafa22ec8d473f31e9"} Oct 30 21:38:11.637828 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637791 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"3ab1a8fb7b394ed5c2ad3173d5294ab9","Type":"ContainerStarted","Data":"3843cedccf2e1add68c02348e56020745d39f4044f652b4cf3350c4ea3ab6c5d"} Oct 30 21:38:11.637828 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637814 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"1d8f9506264df06e876fee4a0052af02","Type":"ContainerStarted","Data":"9fb3d44906776eb889fd1bd8d1a66f05cbcd0680ff184a0711d7f1da997637b4"} Oct 30 21:38:11.637907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637833 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"1d8f9506264df06e876fee4a0052af02","Type":"ContainerDied","Data":"f5217bdad722aafd4341c65a44fc35bf2ed9e33e5e0623f6dce6afcc45e91c62"} Oct 30 21:38:11.637907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637842 5139 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:11.637907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.637853 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" event={"ID":"1d8f9506264df06e876fee4a0052af02","Type":"ContainerStarted","Data":"35ec724587d333a9d8f43a27f5c4330b96293de7b2a9f7870e5f6a90dfaa5a17"} Oct 30 21:38:11.638620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.638600 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:11.638693 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.638624 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:11.638693 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.638635 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:11.638782 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.638754 5139 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-133-5.us-west-2.compute.internal\" not found" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.688947 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.688919 5139 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:11.704572 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.704546 5139 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.731725 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.731700 5139 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 21:38:11.732638 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.732619 5139 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:11.732768 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.732751 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 21:38:11.732815 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.732787 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 21:38:11.732815 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.732797 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeHasSufficientPID" Oct 30 21:38:11.732886 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.732816 5139 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.735755 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.735741 5139 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 21:38:11.736148 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.736133 5139 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:11.779331 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.779301 5139 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 21:38:11.781625 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.781606 5139 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:38:11.781692 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.781653 5139 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.781692 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.781669 5139 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.836191 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.836118 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.836191 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.836155 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ab1a8fb7b394ed5c2ad3173d5294ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"3ab1a8fb7b394ed5c2ad3173d5294ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.836191 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.836180 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.839184 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.839167 5139 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.839245 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.839236 5139 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.839827 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.839801 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeNotReady" Oct 30 21:38:11.839934 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.839831 5139 setters.go:546] "Node became not ready" node="ip-10-0-133-5.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:38:11Z","lastTransitionTime":"2026-10-30T21:38:11Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Oct 30 21:38:11.862670 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.862649 5139 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:38:11.862754 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:11.862688 5139 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.887880 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.887852 5139 nodeinfomanager.go:471] existing CSINode "ip-10-0-133-5.us-west-2.compute.internal" is owned by different node (oldNodeID="4323234f-6646-4714-901a-0a716848777c", newNodeID=""), cleaning up... Oct 30 21:38:11.936916 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.936881 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.936916 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.936913 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.937131 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.936947 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.937131 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.936973 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ab1a8fb7b394ed5c2ad3173d5294ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"3ab1a8fb7b394ed5c2ad3173d5294ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.937131 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.937041 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1d8f9506264df06e876fee4a0052af02-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"1d8f9506264df06e876fee4a0052af02\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:11.937131 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:11.937022 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ab1a8fb7b394ed5c2ad3173d5294ab9-config\") pod \"kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal\" (UID: \"3ab1a8fb7b394ed5c2ad3173d5294ab9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:12.479938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.479907 5139 apiserver.go:52] "Watching apiserver" Oct 30 21:38:12.576698 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.576665 5139 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 21:38:12.577227 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.577203 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7","kube-system/global-pull-secret-syncer-9rfmn","openshift-console/console-55f66b6b69-xpmqp","openshift-dns/node-resolver-ps2p9","openshift-image-registry/image-registry-6f8c95d677-txjhg","openshift-multus/multus-additional-cni-plugins-hw57b","openshift-network-diagnostics/network-check-target-vstcw","openshift-cluster-node-tuning-operator/tuned-hp8q8","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal","openshift-network-operator/iptables-alerter-fb8rx","openshift-ovn-kubernetes/ovnkube-node-2snbw","openshift-insights/insights-runtime-extractor-fcsg4","openshift-ingress-canary/ingress-canary-5gfv2","openshift-monitoring/node-exporter-28r28","openshift-multus/network-metrics-daemon-sbp5h","kube-system/konnectivity-agent-6knhd","openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk","openshift-console/downloads-6dd789c8b7-tvhwl","openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz","openshift-multus/multus-j6hd8","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4","kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal","openshift-image-registry/node-ca-ftmsf","openshift-dns/dns-default-xfjrb"] Oct 30 21:38:12.586722 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.586696 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.588663 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.588646 5139 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:12.588797 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.588782 5139 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:12.605124 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.605106 5139 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 21:38:12.640574 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640551 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-dbus\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:12.640706 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640578 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-bin\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.640706 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640600 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nzqb\" (UniqueName: \"kubernetes.io/projected/114097ab-0272-43e5-a492-f12db369977a-kube-api-access-8nzqb\") pod \"migrator-66df5d496b-4x5sz\" (UID: \"114097ab-0272-43e5-a492-f12db369977a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" Oct 30 21:38:12.640706 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640616 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-wtmp\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.640706 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640665 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w78lc\" (UniqueName: \"kubernetes.io/projected/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-kube-api-access-w78lc\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:38:12.640834 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640812 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-device-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.640884 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640856 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-socket-dir-parent\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.640919 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640892 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mtw\" (UniqueName: \"kubernetes.io/projected/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-kube-api-access-96mtw\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.640919 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640912 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-netns\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.640983 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640958 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.641018 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.640983 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-netd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.641018 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641004 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.641090 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641035 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6jqv\" (UniqueName: \"kubernetes.io/projected/9a49810a-da35-4032-b7ac-a6e3c7453d21-kube-api-access-q6jqv\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:38:12.641090 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641053 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec88c90e-fce8-4f91-9d30-e554174ef776-tmp-dir\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:12.641090 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641082 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpg7q\" (UniqueName: \"kubernetes.io/projected/5230b5e1-f7df-4403-adbb-904ef729b5ab-kube-api-access-hpg7q\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:12.641182 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641133 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ec88c90e-fce8-4f91-9d30-e554174ef776-tmp-dir\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:12.641213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641204 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-var-lib-kubelet\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.641267 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641236 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.641303 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641275 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-kubelet\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.641303 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641290 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-etc-kubernetes\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.641363 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641304 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-conf\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.641363 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641326 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-run\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.641363 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641345 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:12.641476 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641403 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-kubelet-config\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:12.641476 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641451 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysconfig\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.641476 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641470 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cni-binary-copy\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.641569 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641501 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-textfile\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.641569 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641527 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tqq2\" (UniqueName: \"kubernetes.io/projected/5aa67807-4df5-4e6f-a4f9-a378593abf94-kube-api-access-2tqq2\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:38:12.641569 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641544 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-image-registry-private-configuration\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.641569 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641561 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-systemd\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.641687 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641576 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nstvb\" (UniqueName: \"kubernetes.io/projected/d757110b-4b02-4857-853b-949b53441428-kube-api-access-nstvb\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.641687 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641591 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkg86\" (UniqueName: \"kubernetes.io/projected/10088b05-b328-4162-8ef6-39e60f240fb3-kube-api-access-wkg86\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.641687 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641611 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sz7b\" (UniqueName: \"kubernetes.io/projected/261c8437-5164-4544-bd94-90706f1d919b-kube-api-access-2sz7b\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:38:12.641687 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641637 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.641687 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641662 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-certificates\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.641907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641685 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkqn\" (UniqueName: \"kubernetes.io/projected/eab1dc97-e877-4b0f-ab0c-406eb137b296-kube-api-access-nxkqn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.641907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641715 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:12.641907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641740 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-root\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.641907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641756 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-hostroot\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.641907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641781 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42c37929-ba43-4002-877d-a3ab2f5723b8-crio-socket\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.641907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641823 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.641907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641851 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-bound-sa-token\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.641907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641894 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zvr6\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-kube-api-access-8zvr6\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641917 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-kubelet\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641932 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-sys\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641948 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-metrics-client-ca\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641962 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-socket-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.641989 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642035 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovn-node-metrics-cert\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642067 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c8437-5164-4544-bd94-90706f1d919b-cert\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642132 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-system-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642209 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-env-overrides\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642251 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-kubernetes\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.642284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642282 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642310 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/788ef145-30de-43f5-8249-799dbb536b1b-agent-certs\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642368 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-k8s-cni-cncf-io\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642400 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642452 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-trusted-ca\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642480 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-installation-pull-secrets\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642513 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-host\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642539 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-multus\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642583 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-var-lib-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642614 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-config\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642646 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642706 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.642723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642723 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-bin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642740 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-accelerators-collector-config\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642757 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa67807-4df5-4e6f-a4f9-a378593abf94-hosts-file\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642772 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642799 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdq8q\" (UniqueName: \"kubernetes.io/projected/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-kube-api-access-mdq8q\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642819 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642873 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-tmp\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642901 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642931 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a49810a-da35-4032-b7ac-a6e3c7453d21-host\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642955 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-tmp\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.642979 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42c37929-ba43-4002-877d-a3ab2f5723b8-data-volume\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643010 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a776136-2e97-427d-ad80-4b990b59d897-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qrpk7\" (UID: \"9a776136-2e97-427d-ad80-4b990b59d897\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643028 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-systemd-units\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.643098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643093 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmt7z\" (UniqueName: \"kubernetes.io/projected/9b80443a-852a-4e9e-8a6d-26de01cd2912-kube-api-access-kmt7z\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643112 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-tls\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643137 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643166 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/42c37929-ba43-4002-877d-a3ab2f5723b8-data-volume\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643212 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cnibin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643279 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643306 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643340 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlw2c\" (UniqueName: \"kubernetes.io/projected/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-kube-api-access-dlw2c\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643365 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643452 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-registration-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643481 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd9p7\" (UniqueName: \"kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643534 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643569 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.643620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643623 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-modprobe-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643651 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-lib-modules\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643687 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/788ef145-30de-43f5-8249-799dbb536b1b-konnectivity-ca\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643712 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa67807-4df5-4e6f-a4f9-a378593abf94-tmp-dir\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643780 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5aa67807-4df5-4e6f-a4f9-a378593abf94-tmp-dir\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643808 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-multus-certs\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643846 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643869 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643877 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643877 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hkmjr\"" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643901 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5230b5e1-f7df-4403-adbb-904ef729b5ab-tmp\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643906 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643921 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-cnibin\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643949 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88l6l\" (UniqueName: \"kubernetes.io/projected/556b7537-074f-46ce-8db5-e1a0a7b94dca-kube-api-access-88l6l\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643991 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a49810a-da35-4032-b7ac-a6e3c7453d21-serviceca\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.643997 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5230b5e1-f7df-4403-adbb-904ef729b5ab-tmp\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644011 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-daemon-config\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644038 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-tls\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644064 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-script-lib\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.644103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644092 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-host-slash\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644119 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-slash\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644143 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-systemd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644168 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644187 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644202 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-iptables-alerter-script\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644217 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644236 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41712999-d33f-467e-b3b8-dfa8c3fc650b-ca-trust-extracted\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644260 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-sys\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644286 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644306 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-netns\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644323 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr849\" (UniqueName: \"kubernetes.io/projected/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-api-access-kr849\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644341 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-log-socket\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644322 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/41712999-d33f-467e-b3b8-dfa8c3fc650b-ca-trust-extracted\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644365 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-sys-fs\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644388 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-etc-tuned\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644448 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-system-cni-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644509 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644521 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d757110b-4b02-4857-853b-949b53441428-etc-tuned\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644545 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-ovn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644587 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-node-log\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644609 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644624 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-conf-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644651 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-etc-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644680 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec88c90e-fce8-4f91-9d30-e554174ef776-config-volume\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:12.644799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.644706 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzrdr\" (UniqueName: \"kubernetes.io/projected/ec88c90e-fce8-4f91-9d30-e554174ef776-kube-api-access-mzrdr\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:12.652303 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652284 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovn-node-metrics-cert\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.652442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652309 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.652648 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652627 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 21:38:12.652722 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652680 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb4gr\"" Oct 30 21:38:12.652722 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652695 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.652722 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652700 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 21:38:12.652893 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652857 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.652893 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652866 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-config\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.653070 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652921 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.653070 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652934 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 21:38:12.653070 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652942 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.653070 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.652985 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653127 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-phtlh\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653155 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653162 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653276 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653308 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653320 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v9xch\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653330 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j5bph\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653360 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7gkt\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653362 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2fnrw\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653375 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hcvt7\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653380 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ktl9g\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653405 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653384 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 21:38:12.653443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653439 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653516 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653518 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653542 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gfsxs\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653557 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653587 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653593 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653589 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653600 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653592 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653590 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lt5wb\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653712 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653905 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/788ef145-30de-43f5-8249-799dbb536b1b-konnectivity-ca\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653921 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.654087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.653937 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654255 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-daemon-config\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654339 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654388 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654468 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-iptables-alerter-script\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654486 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654474 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654494 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654490 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b80443a-852a-4e9e-8a6d-26de01cd2912-metrics-certs\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654572 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654645 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/843e4020-8588-4ead-adda-ead6c9e0882b-original-pull-secret\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654654 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654747 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.654858 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654861 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9a49810a-da35-4032-b7ac-a6e3c7453d21-serviceca\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:38:12.655264 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.654920 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5bth\"" Oct 30 21:38:12.655264 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.655043 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ec88c90e-fce8-4f91-9d30-e554174ef776-config-volume\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:12.655264 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.655073 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.658758 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.658744 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zlst6\"" Oct 30 21:38:12.658931 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.658915 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b6pf6\"" Oct 30 21:38:12.662138 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.662123 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cni-binary-copy\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.662288 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.662220 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-certificates\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.662288 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.662266 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 21:38:12.662288 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.662277 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/261c8437-5164-4544-bd94-90706f1d919b-cert\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:38:12.662457 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.662341 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlcqp\"" Oct 30 21:38:12.662457 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.662368 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t84mq\"" Oct 30 21:38:12.662457 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.662440 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-env-overrides\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.662625 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.662612 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/788ef145-30de-43f5-8249-799dbb536b1b-agent-certs\") pod \"konnectivity-agent-6knhd\" (UID: \"788ef145-30de-43f5-8249-799dbb536b1b\") " pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:38:12.663553 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.663536 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.663553 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.663547 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/10088b05-b328-4162-8ef6-39e60f240fb3-cni-binary-copy\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.664450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.664432 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 21:38:12.664537 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.664521 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.664650 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.664636 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eab1dc97-e877-4b0f-ab0c-406eb137b296-ovnkube-script-lib\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.671864 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.671847 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ec88c90e-fce8-4f91-9d30-e554174ef776-metrics-tls\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:12.673046 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.673030 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/42c37929-ba43-4002-877d-a3ab2f5723b8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.700530 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.700510 5139 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:38:12.700605 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:12.700547 5139 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:12.701318 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.701303 5139 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 21:38:12.701356 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:12.701341 5139 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-133-5.us-west-2.compute.internal" Oct 30 21:38:12.705632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.705616 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mtw\" (UniqueName: \"kubernetes.io/projected/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-kube-api-access-96mtw\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.709124 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.709107 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tqq2\" (UniqueName: \"kubernetes.io/projected/5aa67807-4df5-4e6f-a4f9-a378593abf94-kube-api-access-2tqq2\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:38:12.710507 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.710491 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6jqv\" (UniqueName: \"kubernetes.io/projected/9a49810a-da35-4032-b7ac-a6e3c7453d21-kube-api-access-q6jqv\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:38:12.717802 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.717773 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 21:38:12.717984 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.717930 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Oct 30 21:38:12.718046 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.718033 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zf5wc\"" Oct 30 21:38:12.718122 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.718050 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-28p6z\"" Oct 30 21:38:12.718122 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.718095 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-g7wm5\"" Oct 30 21:38:12.718222 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.718183 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tsx8f\"" Oct 30 21:38:12.719017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.718959 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 21:38:12.719017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.718968 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 21:38:12.719134 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.719043 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-czbd6\"" Oct 30 21:38:12.719134 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.719091 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 21:38:12.719206 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.719186 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 21:38:12.719337 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.719321 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Oct 30 21:38:12.722535 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.722513 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:12.722613 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.722555 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-29zp4\"" Oct 30 21:38:12.722613 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.722519 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-image-registry-private-configuration\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.723810 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.723794 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.723882 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.723866 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9a776136-2e97-427d-ad80-4b990b59d897-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-qrpk7\" (UID: \"9a776136-2e97-427d-ad80-4b990b59d897\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:12.723924 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.723864 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.724805 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.724789 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-registry-tls\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.724930 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.724913 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:12.725457 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.725439 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr849\" (UniqueName: \"kubernetes.io/projected/42c37929-ba43-4002-877d-a3ab2f5723b8-kube-api-access-kr849\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.725521 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.725442 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w78lc\" (UniqueName: \"kubernetes.io/projected/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-kube-api-access-w78lc\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:38:12.728886 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.728868 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 21:38:12.729227 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.729211 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nstvb\" (UniqueName: \"kubernetes.io/projected/d757110b-4b02-4857-853b-949b53441428-kube-api-access-nstvb\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.729407 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.729392 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62lrs\" (UniqueName: \"kubernetes.io/projected/708926c2-26a2-4888-ae37-a4c157a07e94-kube-api-access-62lrs\") pod \"network-check-target-vstcw\" (UID: \"708926c2-26a2-4888-ae37-a4c157a07e94\") " pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:38:12.729522 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.729508 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-bound-sa-token\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.730535 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.730480 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmt7z\" (UniqueName: \"kubernetes.io/projected/9b80443a-852a-4e9e-8a6d-26de01cd2912-kube-api-access-kmt7z\") pod \"network-metrics-daemon-sbp5h\" (UID: \"9b80443a-852a-4e9e-8a6d-26de01cd2912\") " pod="openshift-multus/network-metrics-daemon-sbp5h" Oct 30 21:38:12.738006 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.737906 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 21:38:12.738006 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.737966 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.738006 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.737975 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.738176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738034 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.738176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738103 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 21:38:12.738176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738167 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 21:38:12.738295 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738212 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.738295 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738286 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 21:38:12.738386 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738369 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 21:38:12.738451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738411 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzrdr\" (UniqueName: \"kubernetes.io/projected/ec88c90e-fce8-4f91-9d30-e554174ef776-kube-api-access-mzrdr\") pod \"dns-default-xfjrb\" (UID: \"ec88c90e-fce8-4f91-9d30-e554174ef776\") " pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:12.738517 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738459 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkqn\" (UniqueName: \"kubernetes.io/projected/eab1dc97-e877-4b0f-ab0c-406eb137b296-kube-api-access-nxkqn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.738517 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738491 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 21:38:12.738645 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738620 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 21:38:12.738874 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738858 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 21:38:12.738943 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738855 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkg86\" (UniqueName: \"kubernetes.io/projected/10088b05-b328-4162-8ef6-39e60f240fb3-kube-api-access-wkg86\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.738943 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738903 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 21:38:12.739017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.738954 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sz7b\" (UniqueName: \"kubernetes.io/projected/261c8437-5164-4544-bd94-90706f1d919b-kube-api-access-2sz7b\") pod \"ingress-canary-5gfv2\" (UID: \"261c8437-5164-4544-bd94-90706f1d919b\") " pod="openshift-ingress-canary/ingress-canary-5gfv2" Oct 30 21:38:12.739058 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.739042 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88l6l\" (UniqueName: \"kubernetes.io/projected/556b7537-074f-46ce-8db5-e1a0a7b94dca-kube-api-access-88l6l\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.739398 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.739378 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zvr6\" (UniqueName: \"kubernetes.io/projected/41712999-d33f-467e-b3b8-dfa8c3fc650b-kube-api-access-8zvr6\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.739496 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.739481 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 21:38:12.741567 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.741550 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpg7q\" (UniqueName: \"kubernetes.io/projected/5230b5e1-f7df-4403-adbb-904ef729b5ab-kube-api-access-hpg7q\") pod \"downloads-6dd789c8b7-tvhwl\" (UID: \"5230b5e1-f7df-4403-adbb-904ef729b5ab\") " pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:12.743444 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.743406 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.743487 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.743427 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.743525 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.743504 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/41712999-d33f-467e-b3b8-dfa8c3fc650b-installation-pull-secrets\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.743597 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.743583 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/41712999-d33f-467e-b3b8-dfa8c3fc650b-trusted-ca\") pod \"image-registry-6f8c95d677-txjhg\" (UID: \"41712999-d33f-467e-b3b8-dfa8c3fc650b\") " pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:12.744626 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.744611 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.744665 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.744622 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:12.745826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745811 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysconfig\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.745870 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745838 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-textfile\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.745870 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745855 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-systemd\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.745939 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745880 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.745939 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745919 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysconfig\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.745939 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745927 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.745939 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745935 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-systemd\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.746061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745967 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-root\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745986 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-hostroot\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.745990 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nzqb\" (UniqueName: \"kubernetes.io/projected/114097ab-0272-43e5-a492-f12db369977a-kube-api-access-8nzqb\") pod \"migrator-66df5d496b-4x5sz\" (UID: \"114097ab-0272-43e5-a492-f12db369977a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-4x5sz" Oct 30 21:38:12.746061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746001 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42c37929-ba43-4002-877d-a3ab2f5723b8-crio-socket\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.746061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746030 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-hostroot\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746029 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-kubelet\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.746061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746055 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-root\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746056 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-sys\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746067 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/42c37929-ba43-4002-877d-a3ab2f5723b8-crio-socket\") pod \"insights-runtime-extractor-fcsg4\" (UID: \"42c37929-ba43-4002-877d-a3ab2f5723b8\") " pod="openshift-insights/insights-runtime-extractor-fcsg4" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746088 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-metrics-client-ca\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746093 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-sys\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746108 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-kubelet\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746110 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-socket-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746143 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-textfile\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746149 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746144 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-socket-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746176 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-system-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746202 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-kubernetes\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746240 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-kubernetes\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746238 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-k8s-cni-cncf-io\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746256 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746267 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-system-cni-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746278 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-host\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746271 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-k8s-cni-cncf-io\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746303 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-multus\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746313 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-host\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746329 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-var-lib-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746342 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-multus\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746358 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-bin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746385 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-accelerators-collector-config\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746410 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-var-lib-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746433 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa67807-4df5-4e6f-a4f9-a378593abf94-hosts-file\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746438 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-cni-bin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746494 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5aa67807-4df5-4e6f-a4f9-a378593abf94-hosts-file\") pod \"node-resolver-ps2p9\" (UID: \"5aa67807-4df5-4e6f-a4f9-a378593abf94\") " pod="openshift-dns/node-resolver-ps2p9" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746503 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746530 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a49810a-da35-4032-b7ac-a6e3c7453d21-host\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746554 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9a49810a-da35-4032-b7ac-a6e3c7453d21-host\") pod \"node-ca-ftmsf\" (UID: \"9a49810a-da35-4032-b7ac-a6e3c7453d21\") " pod="openshift-image-registry/node-ca-ftmsf" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746556 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-systemd-units\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746584 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-systemd-units\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746533 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746593 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-tls\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746620 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cnibin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746653 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746662 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-cnibin\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746655 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-metrics-client-ca\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746749 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlw2c\" (UniqueName: \"kubernetes.io/projected/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-kube-api-access-dlw2c\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.746767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746775 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-registration-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746803 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746816 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-accelerators-collector-config\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746827 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-modprobe-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746853 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-registration-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746867 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-lib-modules\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746889 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-tuning-conf-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746898 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-multus-certs\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746925 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-cnibin\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746935 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-lib-modules\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746937 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-modprobe-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746963 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-cnibin\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746940 5139 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.746981 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-multus-certs\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747018 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-host-slash\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747048 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-slash\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747075 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-systemd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747074 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4d7df9d-6dbd-4b9f-a679-00d8d07db782-host-slash\") pod \"iptables-alerter-fb8rx\" (UID: \"c4d7df9d-6dbd-4b9f-a679-00d8d07db782\") " pod="openshift-network-operator/iptables-alerter-fb8rx" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747110 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-slash\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747121 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-systemd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747135 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-sys\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747152 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-netns\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747171 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-log-socket\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747195 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-sys-fs\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747203 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-sys\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747238 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-run-netns\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747247 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-sys-fs\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747253 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-log-socket\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747271 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-system-cni-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747296 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-ovn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747318 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-node-log\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747341 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747345 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/10088b05-b328-4162-8ef6-39e60f240fb3-system-cni-dir\") pod \"multus-additional-cni-plugins-hw57b\" (UID: \"10088b05-b328-4162-8ef6-39e60f240fb3\") " pod="openshift-multus/multus-additional-cni-plugins-hw57b" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747362 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-ovn\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747362 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-conf-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747387 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-etc-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747389 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-node-log\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747393 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-conf-dir\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747403 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-dbus\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747449 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-bin\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747474 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-etc-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747476 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-wtmp\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747522 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-device-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747555 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-socket-dir-parent\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747573 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-device-dir\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747580 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-wtmp\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747525 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-d\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747590 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-netns\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747617 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-bin\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747631 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-multus-socket-dir-parent\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747666 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747701 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-netd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747667 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-run-netns\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747704 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-run-openvswitch\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.747709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747743 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747758 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-cni-netd\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747774 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-var-lib-kubelet\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747796 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747813 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eab1dc97-e877-4b0f-ab0c-406eb137b296-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-2snbw\" (UID: \"eab1dc97-e877-4b0f-ab0c-406eb137b296\") " pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747817 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-kubelet\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747847 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-host-var-lib-kubelet\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747833 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-var-lib-kubelet\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747864 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-etc-kubernetes\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747877 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/556b7537-074f-46ce-8db5-e1a0a7b94dca-etc-selinux\") pod \"aws-ebs-csi-driver-node-nf5x4\" (UID: \"556b7537-074f-46ce-8db5-e1a0a7b94dca\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nf5x4" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747894 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-conf\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747922 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc-etc-kubernetes\") pod \"multus-j6hd8\" (UID: \"ddc20de6-3c22-4dc4-8b56-9bfd7e9ac9dc\") " pod="openshift-multus/multus-j6hd8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747939 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-run\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747466 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-dbus\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.747965 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-kubelet-config\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.748001 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-run\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.748005 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d757110b-4b02-4857-853b-949b53441428-etc-sysctl-conf\") pod \"tuned-hp8q8\" (UID: \"d757110b-4b02-4857-853b-949b53441428\") " pod="openshift-cluster-node-tuning-operator/tuned-hp8q8" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.748040 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/843e4020-8588-4ead-adda-ead6c9e0882b-kubelet-config\") pod \"global-pull-secret-syncer-9rfmn\" (UID: \"843e4020-8588-4ead-adda-ead6c9e0882b\") " pod="kube-system/global-pull-secret-syncer-9rfmn" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.748553 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd9p7\" (UniqueName: \"kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7\") pod \"console-55f66b6b69-xpmqp\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.748596 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdq8q\" (UniqueName: \"kubernetes.io/projected/fe3f0931-4f7d-4849-9c3d-f09e25aadc18-kube-api-access-mdq8q\") pod \"prometheus-operator-9cd89b74c-z7xtk\" (UID: \"fe3f0931-4f7d-4849-9c3d-f09e25aadc18\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-z7xtk" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.749239 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.749801 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.749517 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-node-exporter-tls\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.805492 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.805458 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlw2c\" (UniqueName: \"kubernetes.io/projected/a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11-kube-api-access-dlw2c\") pod \"node-exporter-28r28\" (UID: \"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11\") " pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.884229 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.884207 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:38:12.887587 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:12.887561 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-28r28" Oct 30 21:38:12.898326 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:12.898284 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e82ff8_c0cc_4e29_a5ba_27c4785b7a11.slice/crio-d1ef7737bbbe756ca1952759a35d21c78aad0bdadac5a177a540c5c07481fbd1 WatchSource:0}: Error finding container d1ef7737bbbe756ca1952759a35d21c78aad0bdadac5a177a540c5c07481fbd1: Status 404 returned error can't find the container with id d1ef7737bbbe756ca1952759a35d21c78aad0bdadac5a177a540c5c07481fbd1 Oct 30 21:38:13.593766 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:13.593727 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28r28" event={"ID":"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11","Type":"ContainerStarted","Data":"d1ef7737bbbe756ca1952759a35d21c78aad0bdadac5a177a540c5c07481fbd1"} Oct 30 21:38:14.245075 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:14.245039 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:14.249789 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:14.249760 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-qrpk7" Oct 30 21:38:14.948878 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:14.948845 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xfjrb" Oct 30 21:38:15.607773 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:15.607735 5139 generic.go:356] "Generic (PLEG): container finished" podID="a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11" containerID="0226b65224eb80f86f55f32386e7bd866a32cb5c3cb4ea56fda695b975132aa3" exitCode=0 Oct 30 21:38:15.607951 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:15.607826 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28r28" event={"ID":"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11","Type":"ContainerDied","Data":"0226b65224eb80f86f55f32386e7bd866a32cb5c3cb4ea56fda695b975132aa3"} Oct 30 21:38:15.996630 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:15.996602 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vstcw" Oct 30 21:38:16.613907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:16.613864 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28r28" event={"ID":"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11","Type":"ContainerStarted","Data":"4c4e45b13048eb7e08a43f72f78bf256863f9ed451be2aae6bf048a33998a25a"} Oct 30 21:38:16.614092 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:16.613939 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-28r28" event={"ID":"a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11","Type":"ContainerStarted","Data":"80344db3871d1c02caa14646aff8579b37d7c1678005b290ee668c2b7d28a14d"} Oct 30 21:38:16.729215 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:16.729160 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-28r28" podStartSLOduration=3.6418056549999998 podStartE2EDuration="5.729140484s" totalImagesPullingTime="2.009937481s" totalInitContainerRuntime="77.397348ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:16.72857018 +0000 UTC m=+5.319855666" watchObservedRunningTime="2026-10-30 21:38:16.729140484 +0000 UTC m=+5.320425972" Oct 30 21:38:17.653269 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:17.653228 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:18.623146 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:18.623107 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5gfv2" event={"ID":"261c8437-5164-4544-bd94-90706f1d919b","Type":"ContainerStarted","Data":"38dcd39c50403e6f97f34a1f9bcf20e1eab566462ea68c4993304c40224314f4"} Oct 30 21:38:19.809827 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:19.809790 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:38:19.889201 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:19.889169 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:20.123096 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:20.123028 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6f8c95d677-txjhg" Oct 30 21:38:20.134240 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:20.134214 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:21.681950 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:21.681852 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f66b6b69-xpmqp"] Oct 30 21:38:21.990638 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:21.990613 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:21.991044 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:21.991021 5139 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 21:38:22.113270 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:22.113230 5139 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-133-5.us-west-2.compute.internal" event="NodeReady" Oct 30 21:38:27.213997 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.213958 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c8dfb96dc-nr26g"] Oct 30 21:38:27.220197 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.220171 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.375635 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.375595 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-service-ca\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.375635 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.375639 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-oauth-serving-cert\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.375854 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.375676 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-config\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.375854 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.375780 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-serving-cert\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.375950 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.375871 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-oauth-config\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.375950 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.375907 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-trusted-ca-bundle\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.375950 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.375937 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c448c13-2f3c-4498-a2b8-674a5d740f57-tmp\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.376092 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.375970 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd8j7\" (UniqueName: \"kubernetes.io/projected/4c448c13-2f3c-4498-a2b8-674a5d740f57-kube-api-access-kd8j7\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.416180 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.416152 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 21:38:27.416962 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.416934 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8dfb96dc-nr26g"] Oct 30 21:38:27.476689 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.476608 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-config\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.476689 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.476662 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-serving-cert\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.476922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.476692 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-oauth-config\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.476922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.476712 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-trusted-ca-bundle\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.476922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.476737 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c448c13-2f3c-4498-a2b8-674a5d740f57-tmp\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.476922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.476764 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd8j7\" (UniqueName: \"kubernetes.io/projected/4c448c13-2f3c-4498-a2b8-674a5d740f57-kube-api-access-kd8j7\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.476922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.476822 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-service-ca\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.476922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.476850 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-oauth-serving-cert\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.477766 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.477736 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c448c13-2f3c-4498-a2b8-674a5d740f57-tmp\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.477898 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.477741 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-oauth-serving-cert\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.478002 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.477962 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-service-ca\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.478117 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.478082 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-trusted-ca-bundle\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.478171 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.478134 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-config\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.480256 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.480235 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-serving-cert\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.480340 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.480310 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-oauth-config\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.551440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.551389 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd8j7\" (UniqueName: \"kubernetes.io/projected/4c448c13-2f3c-4498-a2b8-674a5d740f57-kube-api-access-kd8j7\") pod \"console-c8dfb96dc-nr26g\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:27.834986 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:27.834947 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:28.252749 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:28.252712 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c8dfb96dc-nr26g"] Oct 30 21:38:28.256136 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:28.256093 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c448c13_2f3c_4498_a2b8_674a5d740f57.slice/crio-f0182e85c773d185f1b4629fef89d0633f73fdf95f7672992653375a7cc2d501 WatchSource:0}: Error finding container f0182e85c773d185f1b4629fef89d0633f73fdf95f7672992653375a7cc2d501: Status 404 returned error can't find the container with id f0182e85c773d185f1b4629fef89d0633f73fdf95f7672992653375a7cc2d501 Oct 30 21:38:28.662047 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:28.661967 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8dfb96dc-nr26g" event={"ID":"4c448c13-2f3c-4498-a2b8-674a5d740f57","Type":"ContainerStarted","Data":"bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281"} Oct 30 21:38:28.662047 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:28.662030 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8dfb96dc-nr26g" event={"ID":"4c448c13-2f3c-4498-a2b8-674a5d740f57","Type":"ContainerStarted","Data":"f0182e85c773d185f1b4629fef89d0633f73fdf95f7672992653375a7cc2d501"} Oct 30 21:38:28.788108 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:28.788052 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c8dfb96dc-nr26g" podStartSLOduration=1.788034704 podStartE2EDuration="1.788034704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:28.786355072 +0000 UTC m=+17.377640560" watchObservedRunningTime="2026-10-30 21:38:28.788034704 +0000 UTC m=+17.379320192" Oct 30 21:38:29.010283 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:29.010249 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-2snbw" Oct 30 21:38:30.671407 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:30.671371 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-tvhwl" event={"ID":"5230b5e1-f7df-4403-adbb-904ef729b5ab","Type":"ContainerStarted","Data":"8782f4181d81a7ada5cc8a6618daff5e561568486314bd56775e64db8060c7a1"} Oct 30 21:38:30.671875 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:30.671784 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:30.673407 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:30.673365 5139 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-tvhwl container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" start-of-body= Oct 30 21:38:30.673536 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:30.673456 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-tvhwl" podUID="5230b5e1-f7df-4403-adbb-904ef729b5ab" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.10:8080/\": dial tcp 10.134.0.10:8080: connect: connection refused" Oct 30 21:38:31.693253 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:31.693224 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-tvhwl" Oct 30 21:38:37.835231 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:37.835187 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:37.835809 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:37.835256 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:37.840696 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:37.840671 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:38.705631 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:38.705598 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:38:42.262537 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:42.262504 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6knhd" Oct 30 21:38:46.705682 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:46.705601 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-55f66b6b69-xpmqp" podUID="251a9c03-36b1-463b-bf87-2d831dcd8419" containerName="console" containerID="cri-o://7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9" gracePeriod=15 Oct 30 21:38:46.983111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:46.983040 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:47.038596 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.038556 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp\" (UniqueName: \"kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp\") pod \"251a9c03-36b1-463b-bf87-2d831dcd8419\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " Oct 30 21:38:47.038803 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.038612 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7\" (UniqueName: \"kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7\") pod \"251a9c03-36b1-463b-bf87-2d831dcd8419\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " Oct 30 21:38:47.038803 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.038745 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca\") pod \"251a9c03-36b1-463b-bf87-2d831dcd8419\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " Oct 30 21:38:47.038803 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.038791 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert\") pod \"251a9c03-36b1-463b-bf87-2d831dcd8419\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " Oct 30 21:38:47.038974 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.038819 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config\") pod \"251a9c03-36b1-463b-bf87-2d831dcd8419\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " Oct 30 21:38:47.038974 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.038850 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config\") pod \"251a9c03-36b1-463b-bf87-2d831dcd8419\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " Oct 30 21:38:47.038974 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.038894 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert\") pod \"251a9c03-36b1-463b-bf87-2d831dcd8419\" (UID: \"251a9c03-36b1-463b-bf87-2d831dcd8419\") " Oct 30 21:38:47.038974 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.038893 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp" pod "251a9c03-36b1-463b-bf87-2d831dcd8419" (UID: "251a9c03-36b1-463b-bf87-2d831dcd8419"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:38:47.039211 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.039122 5139 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/251a9c03-36b1-463b-bf87-2d831dcd8419-tmp\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.039211 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.039137 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca" pod "251a9c03-36b1-463b-bf87-2d831dcd8419" (UID: "251a9c03-36b1-463b-bf87-2d831dcd8419"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:47.039282 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.039246 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert" pod "251a9c03-36b1-463b-bf87-2d831dcd8419" (UID: "251a9c03-36b1-463b-bf87-2d831dcd8419"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:47.039282 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.039239 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config" pod "251a9c03-36b1-463b-bf87-2d831dcd8419" (UID: "251a9c03-36b1-463b-bf87-2d831dcd8419"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:38:47.041054 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.041025 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert" pod "251a9c03-36b1-463b-bf87-2d831dcd8419" (UID: "251a9c03-36b1-463b-bf87-2d831dcd8419"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:47.041162 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.041070 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config" pod "251a9c03-36b1-463b-bf87-2d831dcd8419" (UID: "251a9c03-36b1-463b-bf87-2d831dcd8419"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:38:47.041162 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.041123 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7" pod "251a9c03-36b1-463b-bf87-2d831dcd8419" (UID: "251a9c03-36b1-463b-bf87-2d831dcd8419"). InnerVolumeSpecName "kube-api-access-sd9p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:38:47.140493 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.140455 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd9p7\" (UniqueName: \"kubernetes.io/projected/251a9c03-36b1-463b-bf87-2d831dcd8419-kube-api-access-sd9p7\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.140493 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.140490 5139 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.140747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.140503 5139 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-oauth-serving-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.140747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.140518 5139 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/251a9c03-36b1-463b-bf87-2d831dcd8419-console-config\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.140747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.140528 5139 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-oauth-config\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.140747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.140538 5139 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/251a9c03-36b1-463b-bf87-2d831dcd8419-console-serving-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:38:47.729835 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.729802 5139 generic.go:356] "Generic (PLEG): container finished" podID="251a9c03-36b1-463b-bf87-2d831dcd8419" containerID="7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9" exitCode=0 Oct 30 21:38:47.730374 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.729850 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f66b6b69-xpmqp" Oct 30 21:38:47.730374 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.729857 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f66b6b69-xpmqp" event={"ID":"251a9c03-36b1-463b-bf87-2d831dcd8419","Type":"ContainerDied","Data":"7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9"} Oct 30 21:38:47.730374 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.729896 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f66b6b69-xpmqp" event={"ID":"251a9c03-36b1-463b-bf87-2d831dcd8419","Type":"ContainerDied","Data":"4bad963001f4e985be372befd5f053f105d90af6c4af21cf67b44431c53306da"} Oct 30 21:38:47.730374 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.729915 5139 scope.go:122] "RemoveContainer" containerID="7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9" Oct 30 21:38:47.737721 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.737700 5139 scope.go:122] "RemoveContainer" containerID="7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9" Oct 30 21:38:47.737974 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:47.737951 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9\": container with ID starting with 7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9 not found: ID does not exist" containerID="7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9" Oct 30 21:38:47.738042 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.737985 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9"} err="failed to get container status \"7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9\": rpc error: code = NotFound desc = could not find container \"7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9\": container with ID starting with 7ff08c03f75eef89a7e58d02907dc7cf9a647814d79d13e4ef41026256446ef9 not found: ID does not exist" Oct 30 21:38:47.907682 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.907610 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f66b6b69-xpmqp"] Oct 30 21:38:47.923664 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:47.923638 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55f66b6b69-xpmqp"] Oct 30 21:38:49.540005 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:49.539970 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251a9c03-36b1-463b-bf87-2d831dcd8419" path="/var/lib/kubelet/pods/251a9c03-36b1-463b-bf87-2d831dcd8419/volumes" Oct 30 21:38:49.803193 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:38:49.803106 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:38:51.775789 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.775760 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-597f56fb6d-2nbnw"] Oct 30 21:38:51.776213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.775997 5139 cpu_manager.go:414] "containerMap: removing container" podUID="251a9c03-36b1-463b-bf87-2d831dcd8419" containerName="console" Oct 30 21:38:51.776213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.776008 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251a9c03-36b1-463b-bf87-2d831dcd8419" containerName="console" Oct 30 21:38:51.776213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.776058 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="251a9c03-36b1-463b-bf87-2d831dcd8419" containerName="console" Oct 30 21:38:51.798140 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.798116 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.872236 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.872207 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-config\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.872430 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.872243 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-oauth-config\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.872430 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.872266 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-oauth-serving-cert\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.872430 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.872340 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-serving-cert\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.872430 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.872385 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-service-ca\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.872640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.872484 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9c8\" (UniqueName: \"kubernetes.io/projected/f3a1cee2-52a9-4829-aa10-902da9b4b80a-kube-api-access-4s9c8\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.872640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.872530 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3a1cee2-52a9-4829-aa10-902da9b4b80a-tmp\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.872640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.872569 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-trusted-ca-bundle\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.881003 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.880978 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597f56fb6d-2nbnw"] Oct 30 21:38:51.973904 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.973874 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-config\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974106 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.973910 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-oauth-config\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974106 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.973932 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-oauth-serving-cert\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974106 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.973952 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-serving-cert\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974106 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.973969 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-service-ca\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974311 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.974178 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9c8\" (UniqueName: \"kubernetes.io/projected/f3a1cee2-52a9-4829-aa10-902da9b4b80a-kube-api-access-4s9c8\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974311 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.974240 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3a1cee2-52a9-4829-aa10-902da9b4b80a-tmp\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974311 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.974277 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-trusted-ca-bundle\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974663 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.974630 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f3a1cee2-52a9-4829-aa10-902da9b4b80a-tmp\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974776 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.974746 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-service-ca\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.974826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.974790 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-config\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.975002 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.974981 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-oauth-serving-cert\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.975097 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.975080 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3a1cee2-52a9-4829-aa10-902da9b4b80a-trusted-ca-bundle\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.976582 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.976558 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-oauth-config\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:51.976672 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:51.976615 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3a1cee2-52a9-4829-aa10-902da9b4b80a-console-serving-cert\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:52.036948 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:52.036894 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9c8\" (UniqueName: \"kubernetes.io/projected/f3a1cee2-52a9-4829-aa10-902da9b4b80a-kube-api-access-4s9c8\") pod \"console-597f56fb6d-2nbnw\" (UID: \"f3a1cee2-52a9-4829-aa10-902da9b4b80a\") " pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:52.124035 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:52.123991 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:38:52.524534 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:52.524479 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-597f56fb6d-2nbnw"] Oct 30 21:38:52.528954 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:38:52.528923 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a1cee2_52a9_4829_aa10_902da9b4b80a.slice/crio-d2b240066335f54615c13d78b71516a79e85f22270a450c5f1fe0fa2831b2d21 WatchSource:0}: Error finding container d2b240066335f54615c13d78b71516a79e85f22270a450c5f1fe0fa2831b2d21: Status 404 returned error can't find the container with id d2b240066335f54615c13d78b71516a79e85f22270a450c5f1fe0fa2831b2d21 Oct 30 21:38:52.748638 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:52.748600 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597f56fb6d-2nbnw" event={"ID":"f3a1cee2-52a9-4829-aa10-902da9b4b80a","Type":"ContainerStarted","Data":"a494e1402047d00fc411978120e1ba2ef6b76f469f2ef027a1b4cfc70fa94440"} Oct 30 21:38:52.748803 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:52.748657 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-597f56fb6d-2nbnw" event={"ID":"f3a1cee2-52a9-4829-aa10-902da9b4b80a","Type":"ContainerStarted","Data":"d2b240066335f54615c13d78b71516a79e85f22270a450c5f1fe0fa2831b2d21"} Oct 30 21:38:52.868022 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:38:52.867900 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-597f56fb6d-2nbnw" podStartSLOduration=1.867886264 podStartE2EDuration="1.867886264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:52.867716828 +0000 UTC m=+41.459002314" watchObservedRunningTime="2026-10-30 21:38:52.867886264 +0000 UTC m=+41.459171751" Oct 30 21:39:02.124617 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:02.124577 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:39:02.125181 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:02.124658 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:39:02.129353 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:02.129330 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:39:02.785532 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:02.785499 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-597f56fb6d-2nbnw" Oct 30 21:39:02.948191 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:02.948154 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c8dfb96dc-nr26g"] Oct 30 21:39:19.803656 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:39:19.803619 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:39:27.970558 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:27.970513 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c8dfb96dc-nr26g" podUID="4c448c13-2f3c-4498-a2b8-674a5d740f57" containerName="console" containerID="cri-o://bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281" gracePeriod=15 Oct 30 21:39:28.220263 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.220241 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:39:28.360565 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360533 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-config\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-config\") pod \"4c448c13-2f3c-4498-a2b8-674a5d740f57\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " Oct 30 21:39:28.360745 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360602 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-oauth-serving-cert\") pod \"4c448c13-2f3c-4498-a2b8-674a5d740f57\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " Oct 30 21:39:28.360745 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360635 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c448c13-2f3c-4498-a2b8-674a5d740f57-kube-api-access-kd8j7\" (UniqueName: \"kubernetes.io/projected/4c448c13-2f3c-4498-a2b8-674a5d740f57-kube-api-access-kd8j7\") pod \"4c448c13-2f3c-4498-a2b8-674a5d740f57\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " Oct 30 21:39:28.360745 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360664 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-trusted-ca-bundle\") pod \"4c448c13-2f3c-4498-a2b8-674a5d740f57\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " Oct 30 21:39:28.360745 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360718 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-serving-cert\") pod \"4c448c13-2f3c-4498-a2b8-674a5d740f57\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " Oct 30 21:39:28.360970 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360757 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-service-ca\") pod \"4c448c13-2f3c-4498-a2b8-674a5d740f57\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " Oct 30 21:39:28.360970 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360789 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-oauth-config\") pod \"4c448c13-2f3c-4498-a2b8-674a5d740f57\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " Oct 30 21:39:28.360970 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360814 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c448c13-2f3c-4498-a2b8-674a5d740f57-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c448c13-2f3c-4498-a2b8-674a5d740f57-tmp\") pod \"4c448c13-2f3c-4498-a2b8-674a5d740f57\" (UID: \"4c448c13-2f3c-4498-a2b8-674a5d740f57\") " Oct 30 21:39:28.361122 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.360982 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-config" pod "4c448c13-2f3c-4498-a2b8-674a5d740f57" (UID: "4c448c13-2f3c-4498-a2b8-674a5d740f57"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:28.361122 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.361088 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-oauth-serving-cert" pod "4c448c13-2f3c-4498-a2b8-674a5d740f57" (UID: "4c448c13-2f3c-4498-a2b8-674a5d740f57"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:28.361228 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.361190 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c448c13-2f3c-4498-a2b8-674a5d740f57-tmp" pod "4c448c13-2f3c-4498-a2b8-674a5d740f57" (UID: "4c448c13-2f3c-4498-a2b8-674a5d740f57"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:39:28.361266 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.361242 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-trusted-ca-bundle" pod "4c448c13-2f3c-4498-a2b8-674a5d740f57" (UID: "4c448c13-2f3c-4498-a2b8-674a5d740f57"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:28.361299 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.361267 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-service-ca" pod "4c448c13-2f3c-4498-a2b8-674a5d740f57" (UID: "4c448c13-2f3c-4498-a2b8-674a5d740f57"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:39:28.362862 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.362835 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c448c13-2f3c-4498-a2b8-674a5d740f57-kube-api-access-kd8j7" pod "4c448c13-2f3c-4498-a2b8-674a5d740f57" (UID: "4c448c13-2f3c-4498-a2b8-674a5d740f57"). InnerVolumeSpecName "kube-api-access-kd8j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:39:28.362975 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.362920 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-serving-cert" pod "4c448c13-2f3c-4498-a2b8-674a5d740f57" (UID: "4c448c13-2f3c-4498-a2b8-674a5d740f57"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:28.363017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.362967 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-oauth-config" pod "4c448c13-2f3c-4498-a2b8-674a5d740f57" (UID: "4c448c13-2f3c-4498-a2b8-674a5d740f57"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:39:28.461792 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.461752 5139 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-config\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.461792 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.461788 5139 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-oauth-serving-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.461792 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.461797 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd8j7\" (UniqueName: \"kubernetes.io/projected/4c448c13-2f3c-4498-a2b8-674a5d740f57-kube-api-access-kd8j7\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.461792 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.461806 5139 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-trusted-ca-bundle\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.462023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.461815 5139 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-serving-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.462023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.461824 5139 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4c448c13-2f3c-4498-a2b8-674a5d740f57-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.462023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.461832 5139 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4c448c13-2f3c-4498-a2b8-674a5d740f57-console-oauth-config\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.462023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.461841 5139 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4c448c13-2f3c-4498-a2b8-674a5d740f57-tmp\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:39:28.865914 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.865879 5139 generic.go:356] "Generic (PLEG): container finished" podID="4c448c13-2f3c-4498-a2b8-674a5d740f57" containerID="bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281" exitCode=0 Oct 30 21:39:28.866077 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.865925 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c8dfb96dc-nr26g" Oct 30 21:39:28.866077 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.865931 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8dfb96dc-nr26g" event={"ID":"4c448c13-2f3c-4498-a2b8-674a5d740f57","Type":"ContainerDied","Data":"bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281"} Oct 30 21:39:28.866077 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.865973 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c8dfb96dc-nr26g" event={"ID":"4c448c13-2f3c-4498-a2b8-674a5d740f57","Type":"ContainerDied","Data":"f0182e85c773d185f1b4629fef89d0633f73fdf95f7672992653375a7cc2d501"} Oct 30 21:39:28.866077 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.865993 5139 scope.go:122] "RemoveContainer" containerID="bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281" Oct 30 21:39:28.874392 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.874366 5139 scope.go:122] "RemoveContainer" containerID="bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281" Oct 30 21:39:28.874657 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:39:28.874638 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281\": container with ID starting with bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281 not found: ID does not exist" containerID="bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281" Oct 30 21:39:28.874714 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:28.874666 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281"} err="failed to get container status \"bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281\": rpc error: code = NotFound desc = could not find container \"bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281\": container with ID starting with bd19ae01c391fba536150ee8e1a3c0c898930cf3f78241e2068991e0c1e9f281 not found: ID does not exist" Oct 30 21:39:29.023050 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:29.023019 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c8dfb96dc-nr26g"] Oct 30 21:39:29.068985 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:29.068958 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c8dfb96dc-nr26g"] Oct 30 21:39:29.539952 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:39:29.539917 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c448c13-2f3c-4498-a2b8-674a5d740f57" path="/var/lib/kubelet/pods/4c448c13-2f3c-4498-a2b8-674a5d740f57/volumes" Oct 30 21:39:49.804275 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:39:49.804233 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:40:19.802593 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:40:19.802556 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:40:48.156635 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.156605 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh"] Oct 30 21:40:48.157140 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.156851 5139 cpu_manager.go:414] "containerMap: removing container" podUID="4c448c13-2f3c-4498-a2b8-674a5d740f57" containerName="console" Oct 30 21:40:48.157140 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.156861 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c448c13-2f3c-4498-a2b8-674a5d740f57" containerName="console" Oct 30 21:40:48.157140 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.156910 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c448c13-2f3c-4498-a2b8-674a5d740f57" containerName="console" Oct 30 21:40:48.159032 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.159016 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.200501 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.200469 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1944f-0700-4008-a521-d81d4dd3a743-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.200655 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.200517 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.200655 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.200536 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.200655 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.200595 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.200808 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.200717 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.200808 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.200768 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4vw\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-kube-api-access-sl4vw\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.200808 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.200804 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.223341 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.223310 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:40:48.223341 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.223311 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 21:40:48.223523 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.223332 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 21:40:48.264375 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.264350 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 21:40:48.264500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.264353 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 21:40:48.264500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.264394 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 21:40:48.264500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.264424 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-66p9x\"" Oct 30 21:40:48.268986 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.268959 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh"] Oct 30 21:40:48.301701 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.301678 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.301857 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.301707 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4vw\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-kube-api-access-sl4vw\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.301857 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.301735 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.301857 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.301761 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1944f-0700-4008-a521-d81d4dd3a743-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.301857 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.301829 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.302081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.301877 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.302081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.301904 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.302371 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.302338 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.304186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.304166 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1944f-0700-4008-a521-d81d4dd3a743-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.304392 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.304368 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.304492 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.304392 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.304492 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.304462 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.359972 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.359943 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.360276 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.360259 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4vw\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-kube-api-access-sl4vw\") pod \"istiod-openshift-gateway-7876bfcb74-nf5gh\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.469578 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.469503 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:48.880633 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:48.880601 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh"] Oct 30 21:40:48.883889 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:40:48.883849 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc1944f_0700_4008_a521_d81d4dd3a743.slice/crio-a56f71bfb867ece7c5c519785bf59ddb158b232cbe491aa8797b6a05f5033c8c WatchSource:0}: Error finding container a56f71bfb867ece7c5c519785bf59ddb158b232cbe491aa8797b6a05f5033c8c: Status 404 returned error can't find the container with id a56f71bfb867ece7c5c519785bf59ddb158b232cbe491aa8797b6a05f5033c8c Oct 30 21:40:49.132256 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:49.132131 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" event={"ID":"0cc1944f-0700-4008-a521-d81d4dd3a743","Type":"ContainerStarted","Data":"a56f71bfb867ece7c5c519785bf59ddb158b232cbe491aa8797b6a05f5033c8c"} Oct 30 21:40:49.805591 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:40:49.805556 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:40:52.849213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:52.849180 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:52.849473 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:52.849267 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:40:53.148270 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:53.148165 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" event={"ID":"0cc1944f-0700-4008-a521-d81d4dd3a743","Type":"ContainerStarted","Data":"7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638"} Oct 30 21:40:53.148450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:53.148335 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:53.150139 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:53.150112 5139 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-nf5gh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:40:53.150251 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:53.150159 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" podUID="0cc1944f-0700-4008-a521-d81d4dd3a743" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:40:54.152459 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:54.152407 5139 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-nf5gh container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 21:40:54.152925 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:54.152472 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" podUID="0cc1944f-0700-4008-a521-d81d4dd3a743" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 21:40:57.153171 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:57.153139 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:40:57.358870 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:40:57.358813 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" podStartSLOduration=5.396283263 podStartE2EDuration="9.358797749s" totalImagesPullingTime="3.962514486s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:40:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:40:53.34312754 +0000 UTC m=+161.934413027" watchObservedRunningTime="2026-10-30 21:40:57.358797749 +0000 UTC m=+165.950083232" Oct 30 21:41:19.803608 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:41:19.803564 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:41:49.803645 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:41:49.803604 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:42:12.962064 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:12.962030 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58"] Oct 30 21:42:12.964630 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:12.964613 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.017099 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.017065 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.017254 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.017117 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-webhook-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.017254 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.017207 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-metrics-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.017254 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.017230 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-442dg\" (UniqueName: \"kubernetes.io/projected/3c8e95ca-9c57-445a-9a87-0d381c79934f-kube-api-access-442dg\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.017362 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.017256 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c8e95ca-9c57-445a-9a87-0d381c79934f-manifests\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.017362 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.017309 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8e95ca-9c57-445a-9a87-0d381c79934f-tmp\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.074110 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.074078 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 21:42:13.074110 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.074104 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nkddb\"" Oct 30 21:42:13.074290 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.074104 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 21:42:13.074290 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.074104 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 21:42:13.074290 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.074106 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 21:42:13.074290 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.074142 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 21:42:13.078102 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.078078 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58"] Oct 30 21:42:13.117828 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.117798 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-webhook-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.117999 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.117835 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-metrics-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.117999 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.117858 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-442dg\" (UniqueName: \"kubernetes.io/projected/3c8e95ca-9c57-445a-9a87-0d381c79934f-kube-api-access-442dg\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.117999 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.117888 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c8e95ca-9c57-445a-9a87-0d381c79934f-manifests\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.117999 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.117910 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8e95ca-9c57-445a-9a87-0d381c79934f-tmp\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.117999 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.117939 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.118391 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.118366 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3c8e95ca-9c57-445a-9a87-0d381c79934f-manifests\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.120204 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.120184 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3c8e95ca-9c57-445a-9a87-0d381c79934f-tmp\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.120524 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.120497 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-metrics-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.120761 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.120727 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-webhook-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.120830 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.120761 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3c8e95ca-9c57-445a-9a87-0d381c79934f-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.175985 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.175959 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-442dg\" (UniqueName: \"kubernetes.io/projected/3c8e95ca-9c57-445a-9a87-0d381c79934f-kube-api-access-442dg\") pod \"opendatahub-operator-controller-manager-84bc87c977-wpl58\" (UID: \"3c8e95ca-9c57-445a-9a87-0d381c79934f\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.277763 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.277727 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:13.755860 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:13.755789 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58"] Oct 30 21:42:13.759669 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:42:13.759627 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c8e95ca_9c57_445a_9a87_0d381c79934f.slice/crio-edd8b3c37a3e9ea99f8c80a6add9d4a056219a0b5fd67d7ab71ff2e5d8bfbebf WatchSource:0}: Error finding container edd8b3c37a3e9ea99f8c80a6add9d4a056219a0b5fd67d7ab71ff2e5d8bfbebf: Status 404 returned error can't find the container with id edd8b3c37a3e9ea99f8c80a6add9d4a056219a0b5fd67d7ab71ff2e5d8bfbebf Oct 30 21:42:14.412686 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:14.412653 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" event={"ID":"3c8e95ca-9c57-445a-9a87-0d381c79934f","Type":"ContainerStarted","Data":"edd8b3c37a3e9ea99f8c80a6add9d4a056219a0b5fd67d7ab71ff2e5d8bfbebf"} Oct 30 21:42:17.425366 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:17.425335 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" event={"ID":"3c8e95ca-9c57-445a-9a87-0d381c79934f","Type":"ContainerStarted","Data":"b854a5ffe21c0d549a2a1ce0b35910421d4680954c047ad75b7c1b071a6320b3"} Oct 30 21:42:18.430319 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:18.430277 5139 generic.go:356] "Generic (PLEG): container finished" podID="3c8e95ca-9c57-445a-9a87-0d381c79934f" containerID="b854a5ffe21c0d549a2a1ce0b35910421d4680954c047ad75b7c1b071a6320b3" exitCode=0 Oct 30 21:42:18.430755 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:18.430342 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" event={"ID":"3c8e95ca-9c57-445a-9a87-0d381c79934f","Type":"ContainerDied","Data":"b854a5ffe21c0d549a2a1ce0b35910421d4680954c047ad75b7c1b071a6320b3"} Oct 30 21:42:19.435593 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:19.435556 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" event={"ID":"3c8e95ca-9c57-445a-9a87-0d381c79934f","Type":"ContainerStarted","Data":"4f4771877fe47b9bc06d0f7d0656585bc9d1a1c909c680fd9be8006c85acb7aa"} Oct 30 21:42:19.435991 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:19.435682 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:19.583739 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:19.583631 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" podStartSLOduration=2.69410916 podStartE2EDuration="7.583616691s" totalImagesPullingTime="4.419090015s" totalInitContainerRuntime="470.417516ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:19.582139699 +0000 UTC m=+248.173425187" watchObservedRunningTime="2026-10-30 21:42:19.583616691 +0000 UTC m=+248.174902177" Oct 30 21:42:19.803206 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:42:19.803170 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:42:30.441264 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:42:30.441234 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-wpl58" Oct 30 21:42:49.803482 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:42:49.803438 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:43:06.588916 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.588880 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-794f59cf45-94n4w"] Oct 30 21:43:06.591906 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.591887 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.620587 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.620558 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e3b695-4e3c-4992-9485-f1c55585441a-tmp\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.620714 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.620597 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e3b695-4e3c-4992-9485-f1c55585441a-tls-certs\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.620714 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.620672 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhmng\" (UniqueName: \"kubernetes.io/projected/e6e3b695-4e3c-4992-9485-f1c55585441a-kube-api-access-rhmng\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.687101 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.687072 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 21:43:06.687260 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.687168 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 21:43:06.687260 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.687174 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rmm79\"" Oct 30 21:43:06.687975 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.687954 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-794f59cf45-94n4w"] Oct 30 21:43:06.721062 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.721030 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e3b695-4e3c-4992-9485-f1c55585441a-tls-certs\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.721237 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.721078 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhmng\" (UniqueName: \"kubernetes.io/projected/e6e3b695-4e3c-4992-9485-f1c55585441a-kube-api-access-rhmng\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.721237 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.721113 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e3b695-4e3c-4992-9485-f1c55585441a-tmp\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.723357 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.723331 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e6e3b695-4e3c-4992-9485-f1c55585441a-tmp\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.723542 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.723524 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6e3b695-4e3c-4992-9485-f1c55585441a-tls-certs\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.779291 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.779255 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhmng\" (UniqueName: \"kubernetes.io/projected/e6e3b695-4e3c-4992-9485-f1c55585441a-kube-api-access-rhmng\") pod \"kube-auth-proxy-794f59cf45-94n4w\" (UID: \"e6e3b695-4e3c-4992-9485-f1c55585441a\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:06.902975 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:06.902888 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" Oct 30 21:43:07.267451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:07.267404 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-794f59cf45-94n4w"] Oct 30 21:43:07.270672 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:43:07.270645 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6e3b695_4e3c_4992_9485_f1c55585441a.slice/crio-636173a88625025b6012d2c7d01eac03560311dc3e91e2879b7d0ebad78fc189 WatchSource:0}: Error finding container 636173a88625025b6012d2c7d01eac03560311dc3e91e2879b7d0ebad78fc189: Status 404 returned error can't find the container with id 636173a88625025b6012d2c7d01eac03560311dc3e91e2879b7d0ebad78fc189 Oct 30 21:43:07.592832 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:07.592760 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" event={"ID":"e6e3b695-4e3c-4992-9485-f1c55585441a","Type":"ContainerStarted","Data":"636173a88625025b6012d2c7d01eac03560311dc3e91e2879b7d0ebad78fc189"} Oct 30 21:43:09.729787 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.729742 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2"] Oct 30 21:43:09.740057 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.740032 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.740812 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.740784 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d77b1-93d0-403f-855f-6382e8746d5f-metrics-certs\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.740962 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.740823 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e5d77b1-93d0-403f-855f-6382e8746d5f-manager-config\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.740962 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.740878 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ppm2\" (UniqueName: \"kubernetes.io/projected/6e5d77b1-93d0-403f-855f-6382e8746d5f-kube-api-access-4ppm2\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.741076 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.740986 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e5d77b1-93d0-403f-855f-6382e8746d5f-cert\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.789434 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.789387 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2"] Oct 30 21:43:09.834788 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.834751 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 21:43:09.834788 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.834752 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 21:43:09.835014 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.834760 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 21:43:09.836215 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.836192 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 21:43:09.836334 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.836191 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-f47rq\"" Oct 30 21:43:09.836334 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.836197 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 21:43:09.841618 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.841598 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d77b1-93d0-403f-855f-6382e8746d5f-metrics-certs\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.841707 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.841629 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e5d77b1-93d0-403f-855f-6382e8746d5f-manager-config\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.841707 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.841674 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ppm2\" (UniqueName: \"kubernetes.io/projected/6e5d77b1-93d0-403f-855f-6382e8746d5f-kube-api-access-4ppm2\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.841817 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.841724 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e5d77b1-93d0-403f-855f-6382e8746d5f-cert\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.842543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.842519 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6e5d77b1-93d0-403f-855f-6382e8746d5f-manager-config\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.844307 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.844284 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e5d77b1-93d0-403f-855f-6382e8746d5f-metrics-certs\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.844469 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.844451 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6e5d77b1-93d0-403f-855f-6382e8746d5f-cert\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:09.905806 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:09.905741 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ppm2\" (UniqueName: \"kubernetes.io/projected/6e5d77b1-93d0-403f-855f-6382e8746d5f-kube-api-access-4ppm2\") pod \"jobset-controller-manager-75c548b644-v2wl2\" (UID: \"6e5d77b1-93d0-403f-855f-6382e8746d5f\") " pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:10.056336 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:10.056290 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:10.427602 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:10.427566 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2"] Oct 30 21:43:10.431173 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:43:10.431148 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5d77b1_93d0_403f_855f_6382e8746d5f.slice/crio-3ceff71d6e385d086a4b92b0bdbe2010c38895adb02de35159263505e8fbdf0a WatchSource:0}: Error finding container 3ceff71d6e385d086a4b92b0bdbe2010c38895adb02de35159263505e8fbdf0a: Status 404 returned error can't find the container with id 3ceff71d6e385d086a4b92b0bdbe2010c38895adb02de35159263505e8fbdf0a Oct 30 21:43:10.604310 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:10.604220 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" event={"ID":"e6e3b695-4e3c-4992-9485-f1c55585441a","Type":"ContainerStarted","Data":"b8429db9a9ff0b6f7300e1c12bf3066542fb7502c3f0f69318fdf9dc258b6871"} Oct 30 21:43:10.605477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:10.605449 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" event={"ID":"6e5d77b1-93d0-403f-855f-6382e8746d5f","Type":"ContainerStarted","Data":"3ceff71d6e385d086a4b92b0bdbe2010c38895adb02de35159263505e8fbdf0a"} Oct 30 21:43:10.713809 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:10.713743 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-794f59cf45-94n4w" podStartSLOduration=1.694953749 podStartE2EDuration="4.713724968s" totalImagesPullingTime="3.018771219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:10.713233433 +0000 UTC m=+299.304518919" watchObservedRunningTime="2026-10-30 21:43:10.713724968 +0000 UTC m=+299.305010457" Oct 30 21:43:15.626241 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:15.626200 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" event={"ID":"6e5d77b1-93d0-403f-855f-6382e8746d5f","Type":"ContainerStarted","Data":"a5c2d44e4c574cc2bb86a98ed3da5964a9e62d23b8914fb46da80d99ff275ea4"} Oct 30 21:43:15.626693 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:15.626297 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:15.735743 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:15.735572 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" podStartSLOduration=2.395335917 podStartE2EDuration="6.735556072s" totalImagesPullingTime="4.340220155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:15.735221374 +0000 UTC m=+304.326506861" watchObservedRunningTime="2026-10-30 21:43:15.735556072 +0000 UTC m=+304.326841552" Oct 30 21:43:19.804059 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:43:19.804020 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:43:26.637595 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:43:26.637563 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-75c548b644-v2wl2" Oct 30 21:43:49.803350 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:43:49.803306 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:44:19.803361 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:44:19.803323 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:44:28.907726 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:28.907643 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh"] Oct 30 21:44:28.908227 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:28.907844 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" podUID="0cc1944f-0700-4008-a521-d81d4dd3a743" containerName="discovery" containerID="cri-o://7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638" gracePeriod=30 Oct 30 21:44:29.134589 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.134565 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:44:29.242447 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.242404 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-kube-api-access-sl4vw\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-kube-api-access-sl4vw\") pod \"0cc1944f-0700-4008-a521-d81d4dd3a743\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " Oct 30 21:44:29.242591 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.242477 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-ca-configmap\") pod \"0cc1944f-0700-4008-a521-d81d4dd3a743\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " Oct 30 21:44:29.242637 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.242584 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-token\") pod \"0cc1944f-0700-4008-a521-d81d4dd3a743\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " Oct 30 21:44:29.242637 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.242619 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-cacerts\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-cacerts\") pod \"0cc1944f-0700-4008-a521-d81d4dd3a743\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " Oct 30 21:44:29.242744 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.242652 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-dns-cert\") pod \"0cc1944f-0700-4008-a521-d81d4dd3a743\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " Oct 30 21:44:29.242800 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.242754 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cc1944f-0700-4008-a521-d81d4dd3a743-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1944f-0700-4008-a521-d81d4dd3a743-local-certs\") pod \"0cc1944f-0700-4008-a521-d81d4dd3a743\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " Oct 30 21:44:29.242892 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.242856 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-kubeconfig\") pod \"0cc1944f-0700-4008-a521-d81d4dd3a743\" (UID: \"0cc1944f-0700-4008-a521-d81d4dd3a743\") " Oct 30 21:44:29.243129 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.242863 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-ca-configmap" pod "0cc1944f-0700-4008-a521-d81d4dd3a743" (UID: "0cc1944f-0700-4008-a521-d81d4dd3a743"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:44:29.245411 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.245308 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-dns-cert" pod "0cc1944f-0700-4008-a521-d81d4dd3a743" (UID: "0cc1944f-0700-4008-a521-d81d4dd3a743"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:29.245411 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.245319 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-cacerts" pod "0cc1944f-0700-4008-a521-d81d4dd3a743" (UID: "0cc1944f-0700-4008-a521-d81d4dd3a743"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:29.245619 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.245495 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-kube-api-access-sl4vw" pod "0cc1944f-0700-4008-a521-d81d4dd3a743" (UID: "0cc1944f-0700-4008-a521-d81d4dd3a743"). InnerVolumeSpecName "kube-api-access-sl4vw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:29.245619 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.245532 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-token" pod "0cc1944f-0700-4008-a521-d81d4dd3a743" (UID: "0cc1944f-0700-4008-a521-d81d4dd3a743"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:44:29.245619 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.245542 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-kubeconfig" pod "0cc1944f-0700-4008-a521-d81d4dd3a743" (UID: "0cc1944f-0700-4008-a521-d81d4dd3a743"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:44:29.245619 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.245570 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cc1944f-0700-4008-a521-d81d4dd3a743-local-certs" pod "0cc1944f-0700-4008-a521-d81d4dd3a743" (UID: "0cc1944f-0700-4008-a521-d81d4dd3a743"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:44:29.343451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.343396 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:29.343451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.343444 5139 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-cacerts\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:29.343451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.343455 5139 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-dns-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:29.343685 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.343465 5139 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cc1944f-0700-4008-a521-d81d4dd3a743-local-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:29.343685 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.343473 5139 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-kubeconfig\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:29.343685 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.343484 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl4vw\" (UniqueName: \"kubernetes.io/projected/0cc1944f-0700-4008-a521-d81d4dd3a743-kube-api-access-sl4vw\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:29.343685 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.343494 5139 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0cc1944f-0700-4008-a521-d81d4dd3a743-istio-csr-ca-configmap\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:44:29.875541 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.875451 5139 generic.go:356] "Generic (PLEG): container finished" podID="0cc1944f-0700-4008-a521-d81d4dd3a743" containerID="7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638" exitCode=0 Oct 30 21:44:29.875541 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.875488 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" event={"ID":"0cc1944f-0700-4008-a521-d81d4dd3a743","Type":"ContainerDied","Data":"7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638"} Oct 30 21:44:29.875541 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.875510 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" Oct 30 21:44:29.875541 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.875546 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh" event={"ID":"0cc1944f-0700-4008-a521-d81d4dd3a743","Type":"ContainerDied","Data":"a56f71bfb867ece7c5c519785bf59ddb158b232cbe491aa8797b6a05f5033c8c"} Oct 30 21:44:29.875842 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.875567 5139 scope.go:122] "RemoveContainer" containerID="7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638" Oct 30 21:44:29.883375 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.883359 5139 scope.go:122] "RemoveContainer" containerID="7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638" Oct 30 21:44:29.883694 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:44:29.883676 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638\": container with ID starting with 7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638 not found: ID does not exist" containerID="7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638" Oct 30 21:44:29.883760 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:29.883700 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638"} err="failed to get container status \"7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638\": rpc error: code = NotFound desc = could not find container \"7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638\": container with ID starting with 7fc1c411a82bd2356345462ac7d2b82c37852d74fd1990d2c97efc6f32299638 not found: ID does not exist" Oct 30 21:44:30.037864 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:30.037826 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh"] Oct 30 21:44:30.084185 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:30.084146 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-nf5gh"] Oct 30 21:44:31.544856 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:44:31.544821 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc1944f-0700-4008-a521-d81d4dd3a743" path="/var/lib/kubelet/pods/0cc1944f-0700-4008-a521-d81d4dd3a743/volumes" Oct 30 21:44:49.803379 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:44:49.803339 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:45:00.059229 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.059191 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2"] Oct 30 21:45:00.059742 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.059481 5139 cpu_manager.go:414] "containerMap: removing container" podUID="0cc1944f-0700-4008-a521-d81d4dd3a743" containerName="discovery" Oct 30 21:45:00.059742 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.059493 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc1944f-0700-4008-a521-d81d4dd3a743" containerName="discovery" Oct 30 21:45:00.059742 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.059549 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc1944f-0700-4008-a521-d81d4dd3a743" containerName="discovery" Oct 30 21:45:00.062540 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.062520 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.118341 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.118256 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 21:45:00.169396 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.169336 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 21:45:00.169396 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.169359 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zr4d2\"" Oct 30 21:45:00.169396 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.169383 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 21:45:00.169396 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.169353 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 21:45:00.170252 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.170234 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2"] Oct 30 21:45:00.174864 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.174847 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zk4n6\" (UniqueName: \"kubernetes.io/projected/d2a8b152-3a43-45ac-9d65-be439388956a-kube-api-access-zk4n6\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.174921 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.174876 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a8b152-3a43-45ac-9d65-be439388956a-openshift-service-ca\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.174921 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.174903 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-metrics-certs\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.175053 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.175036 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-maas-api-tls\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.276016 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.275981 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-maas-api-tls\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.276181 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.276037 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zk4n6\" (UniqueName: \"kubernetes.io/projected/d2a8b152-3a43-45ac-9d65-be439388956a-kube-api-access-zk4n6\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.276181 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.276059 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a8b152-3a43-45ac-9d65-be439388956a-openshift-service-ca\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.276181 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.276079 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-metrics-certs\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.276857 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.276835 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a8b152-3a43-45ac-9d65-be439388956a-openshift-service-ca\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.278509 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.278488 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-maas-api-tls\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.278606 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.278551 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-metrics-certs\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.338644 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.338582 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zk4n6\" (UniqueName: \"kubernetes.io/projected/d2a8b152-3a43-45ac-9d65-be439388956a-kube-api-access-zk4n6\") pod \"maas-api-746b9bcf78-qkqk2\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.373995 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.373967 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:00.742672 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:45:00.742640 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a8b152_3a43_45ac_9d65_be439388956a.slice/crio-45b3f2c12e9e3bb1fbe3490e7975f973bbf581b1eefd4b12ce9781a39f566f42 WatchSource:0}: Error finding container 45b3f2c12e9e3bb1fbe3490e7975f973bbf581b1eefd4b12ce9781a39f566f42: Status 404 returned error can't find the container with id 45b3f2c12e9e3bb1fbe3490e7975f973bbf581b1eefd4b12ce9781a39f566f42 Oct 30 21:45:00.744343 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.744317 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:45:00.744458 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.744371 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2"] Oct 30 21:45:00.985065 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:00.985025 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" event={"ID":"d2a8b152-3a43-45ac-9d65-be439388956a","Type":"ContainerStarted","Data":"45b3f2c12e9e3bb1fbe3490e7975f973bbf581b1eefd4b12ce9781a39f566f42"} Oct 30 21:45:07.008174 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:07.008132 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" event={"ID":"d2a8b152-3a43-45ac-9d65-be439388956a","Type":"ContainerStarted","Data":"a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c"} Oct 30 21:45:07.008641 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:07.008245 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:07.136335 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:07.136235 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" podStartSLOduration=1.082353783 podStartE2EDuration="7.136220736s" totalImagesPullingTime="6.053866953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:07.136065417 +0000 UTC m=+415.727350916" watchObservedRunningTime="2026-10-30 21:45:07.136220736 +0000 UTC m=+415.727506219" Oct 30 21:45:13.017012 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:13.016982 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:14.083559 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.083528 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4"] Oct 30 21:45:14.087887 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.087868 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.143386 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.143358 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4"] Oct 30 21:45:14.189064 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.189031 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-openshift-service-ca\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.189064 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.189066 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-metrics-certs\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.189256 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.189197 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-maas-api-tls\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.189292 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.189257 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnb9j\" (UniqueName: \"kubernetes.io/projected/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-kube-api-access-wnb9j\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.289672 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.289636 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-maas-api-tls\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.289863 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.289682 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnb9j\" (UniqueName: \"kubernetes.io/projected/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-kube-api-access-wnb9j\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.289863 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.289718 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-openshift-service-ca\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.289863 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.289740 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-metrics-certs\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.290456 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.290433 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-openshift-service-ca\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.292195 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.292176 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-metrics-certs\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.292357 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.292335 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-maas-api-tls\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.354850 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.354779 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnb9j\" (UniqueName: \"kubernetes.io/projected/1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab-kube-api-access-wnb9j\") pod \"maas-api-56c8d9b444-ft4h4\" (UID: \"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab\") " pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.400616 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.400581 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:14.769029 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:45:14.768996 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eeaef3a_f8b8_4502_a419_cad5c2b1d4ab.slice/crio-a14d5756896ff3c82cef94be40b8c8b7e225b5a4de6c3ada8bf5f5d0b5b4f322 WatchSource:0}: Error finding container a14d5756896ff3c82cef94be40b8c8b7e225b5a4de6c3ada8bf5f5d0b5b4f322: Status 404 returned error can't find the container with id a14d5756896ff3c82cef94be40b8c8b7e225b5a4de6c3ada8bf5f5d0b5b4f322 Oct 30 21:45:14.770081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:14.770051 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4"] Oct 30 21:45:15.035490 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:15.035393 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" event={"ID":"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab","Type":"ContainerStarted","Data":"a14d5756896ff3c82cef94be40b8c8b7e225b5a4de6c3ada8bf5f5d0b5b4f322"} Oct 30 21:45:16.040158 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:16.040121 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" event={"ID":"1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab","Type":"ContainerStarted","Data":"5696c11d0a7d9f7cb247787c0d0d7b718d8ce3c41d67e78ef176626e2199e572"} Oct 30 21:45:16.040677 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:16.040258 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:16.163535 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:16.163479 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" podStartSLOduration=1.232475064 podStartE2EDuration="2.163459122s" totalImagesPullingTime="930.984058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:16.162374752 +0000 UTC m=+424.753660238" watchObservedRunningTime="2026-10-30 21:45:16.163459122 +0000 UTC m=+424.754744611" Oct 30 21:45:19.804900 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:45:19.804859 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:45:22.049319 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.049291 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-56c8d9b444-ft4h4" Oct 30 21:45:22.221366 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.221321 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2"] Oct 30 21:45:22.221640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.221600 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" podUID="d2a8b152-3a43-45ac-9d65-be439388956a" containerName="maas-api" containerID="cri-o://a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c" gracePeriod=30 Oct 30 21:45:22.460040 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.460012 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:22.559908 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.559874 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-maas-api-tls\") pod \"d2a8b152-3a43-45ac-9d65-be439388956a\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " Oct 30 21:45:22.560069 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.559958 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a8b152-3a43-45ac-9d65-be439388956a-kube-api-access-zk4n6\" (UniqueName: \"kubernetes.io/projected/d2a8b152-3a43-45ac-9d65-be439388956a-kube-api-access-zk4n6\") pod \"d2a8b152-3a43-45ac-9d65-be439388956a\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " Oct 30 21:45:22.560069 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.560048 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-metrics-certs\") pod \"d2a8b152-3a43-45ac-9d65-be439388956a\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " Oct 30 21:45:22.560184 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.560082 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a8b152-3a43-45ac-9d65-be439388956a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a8b152-3a43-45ac-9d65-be439388956a-openshift-service-ca\") pod \"d2a8b152-3a43-45ac-9d65-be439388956a\" (UID: \"d2a8b152-3a43-45ac-9d65-be439388956a\") " Oct 30 21:45:22.560543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.560513 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a8b152-3a43-45ac-9d65-be439388956a-openshift-service-ca" pod "d2a8b152-3a43-45ac-9d65-be439388956a" (UID: "d2a8b152-3a43-45ac-9d65-be439388956a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:45:22.562079 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.562054 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-maas-api-tls" pod "d2a8b152-3a43-45ac-9d65-be439388956a" (UID: "d2a8b152-3a43-45ac-9d65-be439388956a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:22.562575 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.562557 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a8b152-3a43-45ac-9d65-be439388956a-kube-api-access-zk4n6" pod "d2a8b152-3a43-45ac-9d65-be439388956a" (UID: "d2a8b152-3a43-45ac-9d65-be439388956a"). InnerVolumeSpecName "kube-api-access-zk4n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:45:22.562650 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.562597 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-metrics-certs" pod "d2a8b152-3a43-45ac-9d65-be439388956a" (UID: "d2a8b152-3a43-45ac-9d65-be439388956a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:45:22.660746 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.660657 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.660746 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.660695 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2a8b152-3a43-45ac-9d65-be439388956a-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.660746 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.660706 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d2a8b152-3a43-45ac-9d65-be439388956a-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:22.660746 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:22.660716 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zk4n6\" (UniqueName: \"kubernetes.io/projected/d2a8b152-3a43-45ac-9d65-be439388956a-kube-api-access-zk4n6\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:45:23.066526 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.066485 5139 generic.go:356] "Generic (PLEG): container finished" podID="d2a8b152-3a43-45ac-9d65-be439388956a" containerID="a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c" exitCode=0 Oct 30 21:45:23.066926 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.066536 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" Oct 30 21:45:23.066926 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.066577 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" event={"ID":"d2a8b152-3a43-45ac-9d65-be439388956a","Type":"ContainerDied","Data":"a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c"} Oct 30 21:45:23.066926 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.066635 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2" event={"ID":"d2a8b152-3a43-45ac-9d65-be439388956a","Type":"ContainerDied","Data":"45b3f2c12e9e3bb1fbe3490e7975f973bbf581b1eefd4b12ce9781a39f566f42"} Oct 30 21:45:23.066926 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.066664 5139 scope.go:122] "RemoveContainer" containerID="a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c" Oct 30 21:45:23.075512 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.075492 5139 scope.go:122] "RemoveContainer" containerID="a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c" Oct 30 21:45:23.075793 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:45:23.075773 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c\": container with ID starting with a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c not found: ID does not exist" containerID="a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c" Oct 30 21:45:23.075851 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.075806 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c"} err="failed to get container status \"a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c\": rpc error: code = NotFound desc = could not find container \"a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c\": container with ID starting with a5293f33d5c9a629d4be1b99da918cfa75738ecb974e2e9ad3b325a589e6019c not found: ID does not exist" Oct 30 21:45:23.233161 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.233122 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2"] Oct 30 21:45:23.284450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.284277 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-746b9bcf78-qkqk2"] Oct 30 21:45:23.540669 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:45:23.540634 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a8b152-3a43-45ac-9d65-be439388956a" path="/var/lib/kubelet/pods/d2a8b152-3a43-45ac-9d65-be439388956a/volumes" Oct 30 21:45:49.803316 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:45:49.803273 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:46:13.630746 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.630705 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p5nl5"] Oct 30 21:46:13.631156 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.630974 5139 cpu_manager.go:414] "containerMap: removing container" podUID="d2a8b152-3a43-45ac-9d65-be439388956a" containerName="maas-api" Oct 30 21:46:13.631156 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.630985 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a8b152-3a43-45ac-9d65-be439388956a" containerName="maas-api" Oct 30 21:46:13.631156 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.631043 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a8b152-3a43-45ac-9d65-be439388956a" containerName="maas-api" Oct 30 21:46:13.633732 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.633712 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:13.728746 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.728705 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzctg\" (UniqueName: \"kubernetes.io/projected/05fd7375-474f-4950-adb3-dc0a16c2e215-kube-api-access-wzctg\") pod \"ai-gateway-controller-7d85675f98-p5nl5\" (UID: \"05fd7375-474f-4950-adb3-dc0a16c2e215\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:13.728746 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.728753 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fd7375-474f-4950-adb3-dc0a16c2e215-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p5nl5\" (UID: \"05fd7375-474f-4950-adb3-dc0a16c2e215\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:13.731193 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.731163 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-k72vn\"" Oct 30 21:46:13.731193 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.731189 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 21:46:13.731193 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.731200 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 21:46:13.731393 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.731199 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 21:46:13.731393 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.731261 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 21:46:13.732463 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.732437 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p5nl5"] Oct 30 21:46:13.829579 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.829548 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fd7375-474f-4950-adb3-dc0a16c2e215-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p5nl5\" (UID: \"05fd7375-474f-4950-adb3-dc0a16c2e215\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:13.829800 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.829610 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzctg\" (UniqueName: \"kubernetes.io/projected/05fd7375-474f-4950-adb3-dc0a16c2e215-kube-api-access-wzctg\") pod \"ai-gateway-controller-7d85675f98-p5nl5\" (UID: \"05fd7375-474f-4950-adb3-dc0a16c2e215\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:13.832009 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.831990 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fd7375-474f-4950-adb3-dc0a16c2e215-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-p5nl5\" (UID: \"05fd7375-474f-4950-adb3-dc0a16c2e215\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:13.888443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.888345 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzctg\" (UniqueName: \"kubernetes.io/projected/05fd7375-474f-4950-adb3-dc0a16c2e215-kube-api-access-wzctg\") pod \"ai-gateway-controller-7d85675f98-p5nl5\" (UID: \"05fd7375-474f-4950-adb3-dc0a16c2e215\") " pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:13.944712 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:13.944668 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:14.311026 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:14.310995 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p5nl5"] Oct 30 21:46:14.314180 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:46:14.314146 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05fd7375_474f_4950_adb3_dc0a16c2e215.slice/crio-af0ff18e47274501a1ed101b96f681150dcbc2afc8f8a3755a8b1fcb29b690eb WatchSource:0}: Error finding container af0ff18e47274501a1ed101b96f681150dcbc2afc8f8a3755a8b1fcb29b690eb: Status 404 returned error can't find the container with id af0ff18e47274501a1ed101b96f681150dcbc2afc8f8a3755a8b1fcb29b690eb Oct 30 21:46:15.248276 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:15.248241 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" event={"ID":"05fd7375-474f-4950-adb3-dc0a16c2e215","Type":"ContainerStarted","Data":"af0ff18e47274501a1ed101b96f681150dcbc2afc8f8a3755a8b1fcb29b690eb"} Oct 30 21:46:17.256839 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:17.256798 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" event={"ID":"05fd7375-474f-4950-adb3-dc0a16c2e215","Type":"ContainerStarted","Data":"7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769"} Oct 30 21:46:17.257350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:17.256901 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:17.369486 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:17.369430 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" podStartSLOduration=2.325595764 podStartE2EDuration="4.369401973s" totalImagesPullingTime="2.043806209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:17.367879892 +0000 UTC m=+485.959165379" watchObservedRunningTime="2026-10-30 21:46:17.369401973 +0000 UTC m=+485.960687461" Oct 30 21:46:19.803466 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:46:19.803405 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:46:20.829874 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.829836 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6jftz"] Oct 30 21:46:20.832893 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.832876 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:20.883932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.883898 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e8bb93-0439-4694-b7e8-8a0da249f331-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6jftz\" (UID: \"79e8bb93-0439-4694-b7e8-8a0da249f331\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:20.884103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.883941 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwk7w\" (UniqueName: \"kubernetes.io/projected/79e8bb93-0439-4694-b7e8-8a0da249f331-kube-api-access-fwk7w\") pod \"payload-pre-processing-7b98bb678c-6jftz\" (UID: \"79e8bb93-0439-4694-b7e8-8a0da249f331\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:20.930334 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.930292 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-68g88\"" Oct 30 21:46:20.930515 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.930332 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 21:46:20.931153 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.931135 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6jftz"] Oct 30 21:46:20.977506 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.977467 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ccz7d"] Oct 30 21:46:20.980575 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.980559 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:20.984322 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.984299 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e8bb93-0439-4694-b7e8-8a0da249f331-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6jftz\" (UID: \"79e8bb93-0439-4694-b7e8-8a0da249f331\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:20.984405 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.984351 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwk7w\" (UniqueName: \"kubernetes.io/projected/79e8bb93-0439-4694-b7e8-8a0da249f331-kube-api-access-fwk7w\") pod \"payload-pre-processing-7b98bb678c-6jftz\" (UID: \"79e8bb93-0439-4694-b7e8-8a0da249f331\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:20.985024 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:20.985007 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79e8bb93-0439-4694-b7e8-8a0da249f331-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-6jftz\" (UID: \"79e8bb93-0439-4694-b7e8-8a0da249f331\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:21.047231 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.047197 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ccz7d"] Oct 30 21:46:21.061814 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.061778 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwk7w\" (UniqueName: \"kubernetes.io/projected/79e8bb93-0439-4694-b7e8-8a0da249f331-kube-api-access-fwk7w\") pod \"payload-pre-processing-7b98bb678c-6jftz\" (UID: \"79e8bb93-0439-4694-b7e8-8a0da249f331\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:21.085649 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.085561 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbdcb9f-b9c8-46b3-8925-7c246f2affe3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ccz7d\" (UID: \"9bbdcb9f-b9c8-46b3-8925-7c246f2affe3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:21.085649 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.085612 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrsgw\" (UniqueName: \"kubernetes.io/projected/9bbdcb9f-b9c8-46b3-8925-7c246f2affe3-kube-api-access-vrsgw\") pod \"payload-processing-55fbbbf9dc-ccz7d\" (UID: \"9bbdcb9f-b9c8-46b3-8925-7c246f2affe3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:21.144859 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.144824 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:21.187021 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.186986 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrsgw\" (UniqueName: \"kubernetes.io/projected/9bbdcb9f-b9c8-46b3-8925-7c246f2affe3-kube-api-access-vrsgw\") pod \"payload-processing-55fbbbf9dc-ccz7d\" (UID: \"9bbdcb9f-b9c8-46b3-8925-7c246f2affe3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:21.187190 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.187057 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbdcb9f-b9c8-46b3-8925-7c246f2affe3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ccz7d\" (UID: \"9bbdcb9f-b9c8-46b3-8925-7c246f2affe3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:21.187736 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.187717 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9bbdcb9f-b9c8-46b3-8925-7c246f2affe3-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-ccz7d\" (UID: \"9bbdcb9f-b9c8-46b3-8925-7c246f2affe3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:21.246219 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.246190 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrsgw\" (UniqueName: \"kubernetes.io/projected/9bbdcb9f-b9c8-46b3-8925-7c246f2affe3-kube-api-access-vrsgw\") pod \"payload-processing-55fbbbf9dc-ccz7d\" (UID: \"9bbdcb9f-b9c8-46b3-8925-7c246f2affe3\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:21.292323 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.292284 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:21.510946 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.510908 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-6jftz"] Oct 30 21:46:21.514574 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:46:21.514537 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e8bb93_0439_4694_b7e8_8a0da249f331.slice/crio-5f676c779b43ea401c8e6bc8a00cd73e266cf82c173d08536d46155e8fb158b4 WatchSource:0}: Error finding container 5f676c779b43ea401c8e6bc8a00cd73e266cf82c173d08536d46155e8fb158b4: Status 404 returned error can't find the container with id 5f676c779b43ea401c8e6bc8a00cd73e266cf82c173d08536d46155e8fb158b4 Oct 30 21:46:21.662688 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:21.662581 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-ccz7d"] Oct 30 21:46:21.666533 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:46:21.666505 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bbdcb9f_b9c8_46b3_8925_7c246f2affe3.slice/crio-029b983b0a5d9e35cf8102ddca7a74b7b57646ba10b6541ada3e176750ab5b71 WatchSource:0}: Error finding container 029b983b0a5d9e35cf8102ddca7a74b7b57646ba10b6541ada3e176750ab5b71: Status 404 returned error can't find the container with id 029b983b0a5d9e35cf8102ddca7a74b7b57646ba10b6541ada3e176750ab5b71 Oct 30 21:46:22.274215 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:22.274177 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" event={"ID":"9bbdcb9f-b9c8-46b3-8925-7c246f2affe3","Type":"ContainerStarted","Data":"029b983b0a5d9e35cf8102ddca7a74b7b57646ba10b6541ada3e176750ab5b71"} Oct 30 21:46:22.275277 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:22.275252 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" event={"ID":"79e8bb93-0439-4694-b7e8-8a0da249f331","Type":"ContainerStarted","Data":"5f676c779b43ea401c8e6bc8a00cd73e266cf82c173d08536d46155e8fb158b4"} Oct 30 21:46:25.288907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:25.288862 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" event={"ID":"79e8bb93-0439-4694-b7e8-8a0da249f331","Type":"ContainerStarted","Data":"c16cc6d348073a3beed41bf89669e31209dfec8d609aa07ae604396d9a001882"} Oct 30 21:46:25.289343 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:25.288969 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:25.290371 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:25.290344 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" event={"ID":"9bbdcb9f-b9c8-46b3-8925-7c246f2affe3","Type":"ContainerStarted","Data":"af809dd86c8abcb9ed8c01248dd07d3be6fb35a2760876d096e6d340e5181104"} Oct 30 21:46:25.290506 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:25.290489 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:25.411729 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:25.411676 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" podStartSLOduration=2.571566652 podStartE2EDuration="5.411662999s" totalImagesPullingTime="2.840096347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:25.41047243 +0000 UTC m=+494.001757914" watchObservedRunningTime="2026-10-30 21:46:25.411662999 +0000 UTC m=+494.002948509" Oct 30 21:46:25.521459 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:25.521385 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" podStartSLOduration=2.719811973 podStartE2EDuration="5.521369107s" totalImagesPullingTime="2.801557134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:25.519384186 +0000 UTC m=+494.110669673" watchObservedRunningTime="2026-10-30 21:46:25.521369107 +0000 UTC m=+494.112654643" Oct 30 21:46:28.262970 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.262938 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:28.663189 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.663098 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-75b78c64d-gqk9g"] Oct 30 21:46:28.666525 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.666509 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:28.723444 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.723400 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-75b78c64d-gqk9g"] Oct 30 21:46:28.746806 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.746777 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-579fd\" (UniqueName: \"kubernetes.io/projected/7a96ed85-297f-4508-86bf-f4e6dca54950-kube-api-access-579fd\") pod \"ai-gateway-controller-75b78c64d-gqk9g\" (UID: \"7a96ed85-297f-4508-86bf-f4e6dca54950\") " pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:28.746943 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.746837 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a96ed85-297f-4508-86bf-f4e6dca54950-metrics-certs\") pod \"ai-gateway-controller-75b78c64d-gqk9g\" (UID: \"7a96ed85-297f-4508-86bf-f4e6dca54950\") " pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:28.847765 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.847727 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a96ed85-297f-4508-86bf-f4e6dca54950-metrics-certs\") pod \"ai-gateway-controller-75b78c64d-gqk9g\" (UID: \"7a96ed85-297f-4508-86bf-f4e6dca54950\") " pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:28.847920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.847785 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-579fd\" (UniqueName: \"kubernetes.io/projected/7a96ed85-297f-4508-86bf-f4e6dca54950-kube-api-access-579fd\") pod \"ai-gateway-controller-75b78c64d-gqk9g\" (UID: \"7a96ed85-297f-4508-86bf-f4e6dca54950\") " pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:28.850201 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.850168 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a96ed85-297f-4508-86bf-f4e6dca54950-metrics-certs\") pod \"ai-gateway-controller-75b78c64d-gqk9g\" (UID: \"7a96ed85-297f-4508-86bf-f4e6dca54950\") " pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:28.905949 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.905921 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-579fd\" (UniqueName: \"kubernetes.io/projected/7a96ed85-297f-4508-86bf-f4e6dca54950-kube-api-access-579fd\") pod \"ai-gateway-controller-75b78c64d-gqk9g\" (UID: \"7a96ed85-297f-4508-86bf-f4e6dca54950\") " pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:28.978865 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:28.978784 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:29.349151 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:29.349032 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-75b78c64d-gqk9g"] Oct 30 21:46:29.351905 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:46:29.351863 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a96ed85_297f_4508_86bf_f4e6dca54950.slice/crio-19ddfde8d501327f57875f7f58470a0990423878394224b44eb415dd0266098c WatchSource:0}: Error finding container 19ddfde8d501327f57875f7f58470a0990423878394224b44eb415dd0266098c: Status 404 returned error can't find the container with id 19ddfde8d501327f57875f7f58470a0990423878394224b44eb415dd0266098c Oct 30 21:46:30.310914 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:30.310871 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" event={"ID":"7a96ed85-297f-4508-86bf-f4e6dca54950","Type":"ContainerStarted","Data":"1c76961b7c5026ef77fe8a122293ee73b9d6dba23843c2f7967aba3d94c5f6fe"} Oct 30 21:46:30.311108 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:30.310928 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" event={"ID":"7a96ed85-297f-4508-86bf-f4e6dca54950","Type":"ContainerStarted","Data":"19ddfde8d501327f57875f7f58470a0990423878394224b44eb415dd0266098c"} Oct 30 21:46:30.311108 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:30.311020 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:30.422290 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:30.422115 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" podStartSLOduration=1.5909085269999999 podStartE2EDuration="2.422097938s" totalImagesPullingTime="831.189411ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:30.421474282 +0000 UTC m=+499.012759766" watchObservedRunningTime="2026-10-30 21:46:30.422097938 +0000 UTC m=+499.013383427" Oct 30 21:46:36.298592 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:36.298561 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-ccz7d" Oct 30 21:46:36.299037 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:36.298625 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-6jftz" Oct 30 21:46:41.317498 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.317466 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-75b78c64d-gqk9g" Oct 30 21:46:41.477793 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.477761 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p5nl5"] Oct 30 21:46:41.478090 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.478041 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" podUID="05fd7375-474f-4950-adb3-dc0a16c2e215" containerName="manager" containerID="cri-o://7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769" gracePeriod=30 Oct 30 21:46:41.712576 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.712555 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:41.857930 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.857854 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05fd7375-474f-4950-adb3-dc0a16c2e215-kube-api-access-wzctg\" (UniqueName: \"kubernetes.io/projected/05fd7375-474f-4950-adb3-dc0a16c2e215-kube-api-access-wzctg\") pod \"05fd7375-474f-4950-adb3-dc0a16c2e215\" (UID: \"05fd7375-474f-4950-adb3-dc0a16c2e215\") " Oct 30 21:46:41.857930 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.857891 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05fd7375-474f-4950-adb3-dc0a16c2e215-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fd7375-474f-4950-adb3-dc0a16c2e215-metrics-certs\") pod \"05fd7375-474f-4950-adb3-dc0a16c2e215\" (UID: \"05fd7375-474f-4950-adb3-dc0a16c2e215\") " Oct 30 21:46:41.860000 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.859971 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05fd7375-474f-4950-adb3-dc0a16c2e215-kube-api-access-wzctg" pod "05fd7375-474f-4950-adb3-dc0a16c2e215" (UID: "05fd7375-474f-4950-adb3-dc0a16c2e215"). InnerVolumeSpecName "kube-api-access-wzctg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:46:41.860064 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.860049 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05fd7375-474f-4950-adb3-dc0a16c2e215-metrics-certs" pod "05fd7375-474f-4950-adb3-dc0a16c2e215" (UID: "05fd7375-474f-4950-adb3-dc0a16c2e215"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:46:41.959269 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.959231 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzctg\" (UniqueName: \"kubernetes.io/projected/05fd7375-474f-4950-adb3-dc0a16c2e215-kube-api-access-wzctg\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:41.959269 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:41.959262 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05fd7375-474f-4950-adb3-dc0a16c2e215-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:46:42.357600 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.357566 5139 generic.go:356] "Generic (PLEG): container finished" podID="05fd7375-474f-4950-adb3-dc0a16c2e215" containerID="7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769" exitCode=0 Oct 30 21:46:42.358023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.357611 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" Oct 30 21:46:42.358023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.357636 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" event={"ID":"05fd7375-474f-4950-adb3-dc0a16c2e215","Type":"ContainerDied","Data":"7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769"} Oct 30 21:46:42.358023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.357681 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-p5nl5" event={"ID":"05fd7375-474f-4950-adb3-dc0a16c2e215","Type":"ContainerDied","Data":"af0ff18e47274501a1ed101b96f681150dcbc2afc8f8a3755a8b1fcb29b690eb"} Oct 30 21:46:42.358023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.357704 5139 scope.go:122] "RemoveContainer" containerID="7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769" Oct 30 21:46:42.365704 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.365682 5139 scope.go:122] "RemoveContainer" containerID="7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769" Oct 30 21:46:42.365919 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:46:42.365900 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769\": container with ID starting with 7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769 not found: ID does not exist" containerID="7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769" Oct 30 21:46:42.365976 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.365930 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769"} err="failed to get container status \"7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769\": rpc error: code = NotFound desc = could not find container \"7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769\": container with ID starting with 7f714a512b3983003a6d70eb1350e86e7877a838012ace6dc9ef6d7ed07ed769 not found: ID does not exist" Oct 30 21:46:42.514025 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.513989 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p5nl5"] Oct 30 21:46:42.520338 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:42.520316 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-p5nl5"] Oct 30 21:46:43.540113 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:43.540077 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05fd7375-474f-4950-adb3-dc0a16c2e215" path="/var/lib/kubelet/pods/05fd7375-474f-4950-adb3-dc0a16c2e215/volumes" Oct 30 21:46:49.803510 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:46:49.803469 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:46:51.860335 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.860300 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9"] Oct 30 21:46:51.860733 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.860623 5139 cpu_manager.go:414] "containerMap: removing container" podUID="05fd7375-474f-4950-adb3-dc0a16c2e215" containerName="manager" Oct 30 21:46:51.860733 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.860634 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05fd7375-474f-4950-adb3-dc0a16c2e215" containerName="manager" Oct 30 21:46:51.860733 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.860685 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="05fd7375-474f-4950-adb3-dc0a16c2e215" containerName="manager" Oct 30 21:46:51.866897 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.866873 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:51.942966 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.942931 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9"] Oct 30 21:46:51.975250 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.975220 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 21:46:51.988280 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.988251 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 21:46:51.988280 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.988259 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Oct 30 21:46:51.988391 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:51.988340 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-brncp\"" Oct 30 21:46:52.043515 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.043488 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqwrg\" (UniqueName: \"kubernetes.io/projected/7ab3c454-a14f-40db-ac93-87b411d35f90-kube-api-access-fqwrg\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.043675 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.043533 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.043675 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.043559 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.043780 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.043681 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.043780 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.043748 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.043864 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.043787 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab3c454-a14f-40db-ac93-87b411d35f90-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.144978 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.144890 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.144978 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.144939 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.145214 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.144979 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab3c454-a14f-40db-ac93-87b411d35f90-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.145214 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.145028 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqwrg\" (UniqueName: \"kubernetes.io/projected/7ab3c454-a14f-40db-ac93-87b411d35f90-kube-api-access-fqwrg\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.145214 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.145065 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.145214 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.145096 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.145457 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.145362 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.145457 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.145406 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.145546 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.145480 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.147241 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.147217 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7ab3c454-a14f-40db-ac93-87b411d35f90-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.147388 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.147370 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7ab3c454-a14f-40db-ac93-87b411d35f90-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.210587 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.210556 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqwrg\" (UniqueName: \"kubernetes.io/projected/7ab3c454-a14f-40db-ac93-87b411d35f90-kube-api-access-fqwrg\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9\" (UID: \"7ab3c454-a14f-40db-ac93-87b411d35f90\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.479398 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.479313 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:46:52.856796 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:52.856714 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9"] Oct 30 21:46:52.859060 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:46:52.859030 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab3c454_a14f_40db_ac93_87b411d35f90.slice/crio-081b2daad071da47d74944fb1058a130cfd8641c2a0db671706b78a39a1b6246 WatchSource:0}: Error finding container 081b2daad071da47d74944fb1058a130cfd8641c2a0db671706b78a39a1b6246: Status 404 returned error can't find the container with id 081b2daad071da47d74944fb1058a130cfd8641c2a0db671706b78a39a1b6246 Oct 30 21:46:53.401928 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:53.401892 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" event={"ID":"7ab3c454-a14f-40db-ac93-87b411d35f90","Type":"ContainerStarted","Data":"081b2daad071da47d74944fb1058a130cfd8641c2a0db671706b78a39a1b6246"} Oct 30 21:46:55.412503 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:55.412464 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" event={"ID":"7ab3c454-a14f-40db-ac93-87b411d35f90","Type":"ContainerStarted","Data":"9123541a7a60e6cd53f3f434b5464804927e58224b6c74d18e4feba651e82976"} Oct 30 21:46:55.479986 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:46:55.479954 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:47:02.078629 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.078571 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" podStartSLOduration=9.170034174 podStartE2EDuration="11.078552682s" totalImagesPullingTime="1.908518508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:46:55.528274873 +0000 UTC m=+524.119560359" watchObservedRunningTime="2026-10-30 21:47:02.078552682 +0000 UTC m=+530.669838169" Oct 30 21:47:02.079065 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.079047 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q"] Oct 30 21:47:02.082376 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.082359 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.120048 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.120007 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.120186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.120075 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8dj7\" (UniqueName: \"kubernetes.io/projected/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-kube-api-access-p8dj7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.120186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.120146 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.120271 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.120246 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.120315 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.120296 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.120353 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.120339 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.178011 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.177965 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 21:47:02.178994 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.178970 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q"] Oct 30 21:47:02.221439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.221376 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.221640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.221493 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.221640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.221527 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.221640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.221551 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.221640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.221582 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.221640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.221615 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8dj7\" (UniqueName: \"kubernetes.io/projected/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-kube-api-access-p8dj7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.221926 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.221897 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.222049 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.221960 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.222110 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.222083 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.223734 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.223715 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.224095 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.224075 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.279996 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.279958 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8dj7\" (UniqueName: \"kubernetes.io/projected/99d60883-524c-4d1b-a77a-2ffd15d3fbfd-kube-api-access-p8dj7\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6b74q\" (UID: \"99d60883-524c-4d1b-a77a-2ffd15d3fbfd\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.394190 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.394096 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:02.479876 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.479843 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:47:02.492816 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.492786 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:47:02.760888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:02.760859 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q"] Oct 30 21:47:02.764011 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:47:02.763985 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d60883_524c_4d1b_a77a_2ffd15d3fbfd.slice/crio-844433a8df64caf3561e712380efff90db6bb16e7a80fb650d55b985d2068de1 WatchSource:0}: Error finding container 844433a8df64caf3561e712380efff90db6bb16e7a80fb650d55b985d2068de1: Status 404 returned error can't find the container with id 844433a8df64caf3561e712380efff90db6bb16e7a80fb650d55b985d2068de1 Oct 30 21:47:03.445240 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:03.445200 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" event={"ID":"99d60883-524c-4d1b-a77a-2ffd15d3fbfd","Type":"ContainerStarted","Data":"d36368643cdab77f34c39f6df5e4860863b6b599fcc5588b3b70df5fec82992a"} Oct 30 21:47:03.445677 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:03.445268 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" event={"ID":"99d60883-524c-4d1b-a77a-2ffd15d3fbfd","Type":"ContainerStarted","Data":"844433a8df64caf3561e712380efff90db6bb16e7a80fb650d55b985d2068de1"} Oct 30 21:47:03.457322 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:03.457298 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9" Oct 30 21:47:03.560788 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:03.560723 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" podStartSLOduration=0.984451596 podStartE2EDuration="1.560697326s" totalImagesPullingTime="576.24573ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:03.559320128 +0000 UTC m=+532.150605614" watchObservedRunningTime="2026-10-30 21:47:03.560697326 +0000 UTC m=+532.151982814" Oct 30 21:47:04.394637 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:04.394584 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:05.512743 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.512702 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz"] Oct 30 21:47:05.516134 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.516113 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.549826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.549791 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.549962 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.549845 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.550040 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.549957 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.550040 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.549992 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8587fc1-8061-4c87-913c-8f3f08a3108d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.550160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.550048 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjn9b\" (UniqueName: \"kubernetes.io/projected/a8587fc1-8061-4c87-913c-8f3f08a3108d-kube-api-access-tjn9b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.550160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.550099 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.582179 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.582155 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 21:47:05.614758 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.614732 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz"] Oct 30 21:47:05.650946 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.650920 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.651080 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.650979 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.651080 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.651004 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8587fc1-8061-4c87-913c-8f3f08a3108d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.651080 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.651024 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjn9b\" (UniqueName: \"kubernetes.io/projected/a8587fc1-8061-4c87-913c-8f3f08a3108d-kube-api-access-tjn9b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.651240 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.651076 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.651240 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.651126 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.651458 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.651410 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.651458 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.651451 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.651727 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.651699 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.653385 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.653365 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8587fc1-8061-4c87-913c-8f3f08a3108d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.653793 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.653771 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8587fc1-8061-4c87-913c-8f3f08a3108d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.718108 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.718073 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjn9b\" (UniqueName: \"kubernetes.io/projected/a8587fc1-8061-4c87-913c-8f3f08a3108d-kube-api-access-tjn9b\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-phngz\" (UID: \"a8587fc1-8061-4c87-913c-8f3f08a3108d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:05.828938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:05.828847 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:06.199026 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:06.198904 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz"] Oct 30 21:47:06.201729 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:47:06.201701 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8587fc1_8061_4c87_913c_8f3f08a3108d.slice/crio-cd0e1f58ebce1a8101056a3c87af0f26f543c232dfbff073b746ecb09351b7c7 WatchSource:0}: Error finding container cd0e1f58ebce1a8101056a3c87af0f26f543c232dfbff073b746ecb09351b7c7: Status 404 returned error can't find the container with id cd0e1f58ebce1a8101056a3c87af0f26f543c232dfbff073b746ecb09351b7c7 Oct 30 21:47:06.462845 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:06.462763 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" event={"ID":"a8587fc1-8061-4c87-913c-8f3f08a3108d","Type":"ContainerStarted","Data":"cd0e1f58ebce1a8101056a3c87af0f26f543c232dfbff073b746ecb09351b7c7"} Oct 30 21:47:07.468123 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:07.468085 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" event={"ID":"a8587fc1-8061-4c87-913c-8f3f08a3108d","Type":"ContainerStarted","Data":"c2ca1be5142542b8d2993b229498d83f7ccec0b78f6078cd40833549383dbfdb"} Oct 30 21:47:07.584830 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:07.584762 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" podStartSLOduration=2.040998378 podStartE2EDuration="2.584746817s" totalImagesPullingTime="543.748439ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:07.584310913 +0000 UTC m=+536.175596400" watchObservedRunningTime="2026-10-30 21:47:07.584746817 +0000 UTC m=+536.176032356" Oct 30 21:47:07.828979 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:07.828946 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:08.497449 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.497391 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn"] Oct 30 21:47:08.500791 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.500772 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.563329 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.563294 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn"] Oct 30 21:47:08.576199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.576169 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.576344 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.576208 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.576344 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.576229 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.576344 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.576250 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bb43f0-83cd-4027-9870-1b48949a2ef2-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.576503 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.576347 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.576503 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.576439 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rt8g\" (UniqueName: \"kubernetes.io/projected/40bb43f0-83cd-4027-9870-1b48949a2ef2-kube-api-access-6rt8g\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.608391 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.608362 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-external-token-limit-simulated-kserve-self-signed-certs\"" Oct 30 21:47:08.677919 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.677881 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bb43f0-83cd-4027-9870-1b48949a2ef2-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.677919 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.677920 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.678176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.677943 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rt8g\" (UniqueName: \"kubernetes.io/projected/40bb43f0-83cd-4027-9870-1b48949a2ef2-kube-api-access-6rt8g\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.678176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.678029 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.678176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.678071 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.678176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.678096 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.678446 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.678404 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-model-cache\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.678511 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.678448 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-home\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.678549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.678530 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-tmp-dir\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.680331 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.680304 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/40bb43f0-83cd-4027-9870-1b48949a2ef2-dshm\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.680733 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.680704 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/40bb43f0-83cd-4027-9870-1b48949a2ef2-tls-certs\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.736839 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.736802 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rt8g\" (UniqueName: \"kubernetes.io/projected/40bb43f0-83cd-4027-9870-1b48949a2ef2-kube-api-access-6rt8g\") pod \"e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn\" (UID: \"40bb43f0-83cd-4027-9870-1b48949a2ef2\") " pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:08.812856 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:08.812763 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:09.178549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:09.178468 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn"] Oct 30 21:47:09.181272 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:47:09.181244 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bb43f0_83cd_4027_9870_1b48949a2ef2.slice/crio-cae4b32643c183a932b5b94bef3c4a3c10f5ace4cb8d5368719ba91653ede4f5 WatchSource:0}: Error finding container cae4b32643c183a932b5b94bef3c4a3c10f5ace4cb8d5368719ba91653ede4f5: Status 404 returned error can't find the container with id cae4b32643c183a932b5b94bef3c4a3c10f5ace4cb8d5368719ba91653ede4f5 Oct 30 21:47:09.477531 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:09.477446 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" event={"ID":"40bb43f0-83cd-4027-9870-1b48949a2ef2","Type":"ContainerStarted","Data":"cae4b32643c183a932b5b94bef3c4a3c10f5ace4cb8d5368719ba91653ede4f5"} Oct 30 21:47:10.483845 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:10.483807 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" event={"ID":"40bb43f0-83cd-4027-9870-1b48949a2ef2","Type":"ContainerStarted","Data":"4289cdc6bec3201c54962e4bf0685bbe482ed6bf46f99544a3aada50bd75bf46"} Oct 30 21:47:10.595067 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:10.594993 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" podStartSLOduration=2.033320584 podStartE2EDuration="2.594980391s" totalImagesPullingTime="561.659807ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:10.594804569 +0000 UTC m=+539.186090055" watchObservedRunningTime="2026-10-30 21:47:10.594980391 +0000 UTC m=+539.186265877" Oct 30 21:47:10.813673 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:10.813637 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:11.901932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:11.901852 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf"] Oct 30 21:47:11.906406 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:11.906382 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.004563 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.004532 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 21:47:12.005859 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.005825 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.006008 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.005879 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.006008 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.005926 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf"] Oct 30 21:47:12.006093 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.006008 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.006093 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.006034 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9284508-ee62-4a92-bcef-88e7cf77f265-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.006093 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.006064 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwz9r\" (UniqueName: \"kubernetes.io/projected/d9284508-ee62-4a92-bcef-88e7cf77f265-kube-api-access-kwz9r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.006197 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.006093 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107231 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107190 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107475 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107251 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107475 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107282 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107475 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107306 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9284508-ee62-4a92-bcef-88e7cf77f265-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107475 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107339 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwz9r\" (UniqueName: \"kubernetes.io/projected/d9284508-ee62-4a92-bcef-88e7cf77f265-kube-api-access-kwz9r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107475 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107373 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107753 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107708 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107753 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107730 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.107869 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.107852 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.109660 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.109640 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9284508-ee62-4a92-bcef-88e7cf77f265-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.109842 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.109822 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9284508-ee62-4a92-bcef-88e7cf77f265-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.166043 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.165957 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwz9r\" (UniqueName: \"kubernetes.io/projected/d9284508-ee62-4a92-bcef-88e7cf77f265-kube-api-access-kwz9r\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-zsvqf\" (UID: \"d9284508-ee62-4a92-bcef-88e7cf77f265\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.218056 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.218016 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:12.395192 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.395151 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:12.408057 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.408027 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:12.502814 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.502782 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6b74q" Oct 30 21:47:12.633507 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:12.633470 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf"] Oct 30 21:47:12.636383 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:47:12.636353 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9284508_ee62_4a92_bcef_88e7cf77f265.slice/crio-f76188cf3f363f2e9cf50077c80cd906c89a729f6f51b6467be637efc725482b WatchSource:0}: Error finding container f76188cf3f363f2e9cf50077c80cd906c89a729f6f51b6467be637efc725482b: Status 404 returned error can't find the container with id f76188cf3f363f2e9cf50077c80cd906c89a729f6f51b6467be637efc725482b Oct 30 21:47:13.497486 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:13.497444 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" event={"ID":"d9284508-ee62-4a92-bcef-88e7cf77f265","Type":"ContainerStarted","Data":"4265359df4c3ae569ec65c5eb16e71ac94be23d9b7ccacbbf6f06cc307724b0b"} Oct 30 21:47:13.497955 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:13.497549 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" event={"ID":"d9284508-ee62-4a92-bcef-88e7cf77f265","Type":"ContainerStarted","Data":"f76188cf3f363f2e9cf50077c80cd906c89a729f6f51b6467be637efc725482b"} Oct 30 21:47:13.615932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:13.615875 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" podStartSLOduration=2.023818837 podStartE2EDuration="2.615858677s" totalImagesPullingTime="592.03984ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:13.614908952 +0000 UTC m=+542.206194449" watchObservedRunningTime="2026-10-30 21:47:13.615858677 +0000 UTC m=+542.207144163" Oct 30 21:47:14.218966 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:14.218928 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:15.643053 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.643019 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr"] Oct 30 21:47:15.647819 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.647792 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.698772 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.698744 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 21:47:15.704200 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.704171 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr"] Oct 30 21:47:15.739387 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.739359 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.739555 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.739403 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frqlh\" (UniqueName: \"kubernetes.io/projected/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-kube-api-access-frqlh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.739555 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.739508 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.739555 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.739536 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.739668 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.739648 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.739704 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.739680 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.829493 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.829462 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:15.840474 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.840448 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frqlh\" (UniqueName: \"kubernetes.io/projected/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-kube-api-access-frqlh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.840623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.840495 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.840623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.840514 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.840623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.840566 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.840623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.840590 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.840623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.840620 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.840965 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.840933 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.841107 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.841026 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.841107 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.840997 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.842199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.842183 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:15.842815 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.842796 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.843113 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.843091 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.898588 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.898522 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frqlh\" (UniqueName: \"kubernetes.io/projected/d78c88db-4ef9-4d16-b3ed-9d8f4b291470-kube-api-access-frqlh\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr\" (UID: \"d78c88db-4ef9-4d16-b3ed-9d8f4b291470\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:15.960236 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:15.960206 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:16.327457 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:16.327425 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr"] Oct 30 21:47:16.330836 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:47:16.330798 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd78c88db_4ef9_4d16_b3ed_9d8f4b291470.slice/crio-4a472972583a004a565176a3f4b7fc458cb2ba04debc2060664f01ba0662fbc4 WatchSource:0}: Error finding container 4a472972583a004a565176a3f4b7fc458cb2ba04debc2060664f01ba0662fbc4: Status 404 returned error can't find the container with id 4a472972583a004a565176a3f4b7fc458cb2ba04debc2060664f01ba0662fbc4 Oct 30 21:47:16.510054 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:16.510015 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" event={"ID":"d78c88db-4ef9-4d16-b3ed-9d8f4b291470","Type":"ContainerStarted","Data":"4a472972583a004a565176a3f4b7fc458cb2ba04debc2060664f01ba0662fbc4"} Oct 30 21:47:16.521114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:16.521089 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-phngz" Oct 30 21:47:17.516556 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:17.516520 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" event={"ID":"d78c88db-4ef9-4d16-b3ed-9d8f4b291470","Type":"ContainerStarted","Data":"55f739030cadaef2f1124b4cf88051e972a5ae185fa4d6586d73921185a39aff"} Oct 30 21:47:17.630249 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:17.630195 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" podStartSLOduration=2.071047294 podStartE2EDuration="2.630180254s" totalImagesPullingTime="559.13296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:17.629370461 +0000 UTC m=+546.220655952" watchObservedRunningTime="2026-10-30 21:47:17.630180254 +0000 UTC m=+546.221465763" Oct 30 21:47:17.961108 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:17.961027 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:18.813604 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:18.813564 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:18.828446 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:18.828405 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:19.540807 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:19.540779 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn" Oct 30 21:47:19.804689 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:47:19.804596 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:47:22.219232 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:22.219148 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:22.231613 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:22.231586 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:22.549861 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:22.549828 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-zsvqf" Oct 30 21:47:23.023994 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.023957 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6"] Oct 30 21:47:23.027613 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.027594 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.079818 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.079786 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 21:47:23.084893 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.084841 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6"] Oct 30 21:47:23.106847 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.106818 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.107066 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.106858 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.107066 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.106879 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/238c2b96-8487-422e-8fd3-9a757fe2169d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.107066 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.107015 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.107208 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.107130 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt688\" (UniqueName: \"kubernetes.io/projected/238c2b96-8487-422e-8fd3-9a757fe2169d-kube-api-access-bt688\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.107258 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.107227 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.207765 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.207726 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.207765 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.207766 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.208017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.207782 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/238c2b96-8487-422e-8fd3-9a757fe2169d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.208017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.207810 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.208017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.207852 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt688\" (UniqueName: \"kubernetes.io/projected/238c2b96-8487-422e-8fd3-9a757fe2169d-kube-api-access-bt688\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.208017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.207988 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.208222 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.208184 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.208222 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.208194 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.208320 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.208311 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.210147 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.210123 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/238c2b96-8487-422e-8fd3-9a757fe2169d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.210332 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.210315 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/238c2b96-8487-422e-8fd3-9a757fe2169d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.266158 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.266121 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt688\" (UniqueName: \"kubernetes.io/projected/238c2b96-8487-422e-8fd3-9a757fe2169d-kube-api-access-bt688\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-564c6\" (UID: \"238c2b96-8487-422e-8fd3-9a757fe2169d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.339783 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.339705 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:23.705101 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:23.705019 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6"] Oct 30 21:47:23.707944 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:47:23.707915 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod238c2b96_8487_422e_8fd3_9a757fe2169d.slice/crio-02186dae3e3d4ba71f5b7ec4ed34c6a88c245f6fc9201bfb39da120ec91bc76d WatchSource:0}: Error finding container 02186dae3e3d4ba71f5b7ec4ed34c6a88c245f6fc9201bfb39da120ec91bc76d: Status 404 returned error can't find the container with id 02186dae3e3d4ba71f5b7ec4ed34c6a88c245f6fc9201bfb39da120ec91bc76d Oct 30 21:47:24.547633 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:24.547598 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" event={"ID":"238c2b96-8487-422e-8fd3-9a757fe2169d","Type":"ContainerStarted","Data":"123fcb3bb9322f7cada23c3a5875c26220f966dda3b48bec7c9143a64c9e950d"} Oct 30 21:47:24.548005 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:24.547672 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" event={"ID":"238c2b96-8487-422e-8fd3-9a757fe2169d","Type":"ContainerStarted","Data":"02186dae3e3d4ba71f5b7ec4ed34c6a88c245f6fc9201bfb39da120ec91bc76d"} Oct 30 21:47:24.667951 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:24.667830 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" podStartSLOduration=2.067438984 podStartE2EDuration="2.667815874s" totalImagesPullingTime="600.37689ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:24.667146641 +0000 UTC m=+553.258432127" watchObservedRunningTime="2026-10-30 21:47:24.667815874 +0000 UTC m=+553.259101365" Oct 30 21:47:25.340126 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:25.340093 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:25.961359 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:25.961314 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:25.974223 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:25.974193 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:26.217957 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.217878 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl"] Oct 30 21:47:26.221593 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.221575 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.271008 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.270985 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 21:47:26.278021 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.277993 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl"] Oct 30 21:47:26.335541 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.335509 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.335723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.335564 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.335723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.335703 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.335820 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.335742 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.335820 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.335760 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cnnb\" (UniqueName: \"kubernetes.io/projected/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-kube-api-access-9cnnb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.335820 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.335784 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437298 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437264 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437462 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437309 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437462 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437343 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437458 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437501 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437522 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cnnb\" (UniqueName: \"kubernetes.io/projected/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-kube-api-access-9cnnb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437772 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437752 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437821 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437773 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.437922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.437901 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.439659 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.439638 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.439909 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.439890 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.507660 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.507630 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cnnb\" (UniqueName: \"kubernetes.io/projected/35879e2e-b0ac-4f01-ab06-79ace61c7f2b-kube-api-access-9cnnb\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl\" (UID: \"35879e2e-b0ac-4f01-ab06-79ace61c7f2b\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.554697 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.554664 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:26.567689 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.567668 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr" Oct 30 21:47:26.921456 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:26.921357 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl"] Oct 30 21:47:26.924935 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:47:26.924903 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35879e2e_b0ac_4f01_ab06_79ace61c7f2b.slice/crio-0a14ba0e28cf092c3af04a989e2e530c195b9610e7e57a954923a7424da9a89b WatchSource:0}: Error finding container 0a14ba0e28cf092c3af04a989e2e530c195b9610e7e57a954923a7424da9a89b: Status 404 returned error can't find the container with id 0a14ba0e28cf092c3af04a989e2e530c195b9610e7e57a954923a7424da9a89b Oct 30 21:47:27.562594 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:27.562563 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" event={"ID":"35879e2e-b0ac-4f01-ab06-79ace61c7f2b","Type":"ContainerStarted","Data":"0a14ba0e28cf092c3af04a989e2e530c195b9610e7e57a954923a7424da9a89b"} Oct 30 21:47:28.568774 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:28.568736 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" event={"ID":"35879e2e-b0ac-4f01-ab06-79ace61c7f2b","Type":"ContainerStarted","Data":"f440db16a16addc8e5efd133fd37cdde78cc4eedae35e5ee024eb597db011da5"} Oct 30 21:47:28.693221 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:28.693169 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podStartSLOduration=2.1190172 podStartE2EDuration="2.693156478s" totalImagesPullingTime="574.139278ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:28.692658122 +0000 UTC m=+557.283943609" watchObservedRunningTime="2026-10-30 21:47:28.693156478 +0000 UTC m=+557.284441966" Oct 30 21:47:29.555216 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:29.555175 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:33.340862 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:33.340817 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:33.353004 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:33.352976 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:33.600495 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:33.600404 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-564c6" Oct 30 21:47:36.554912 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:36.554874 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:36.568136 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:36.568111 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:36.615518 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:36.615488 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:47:44.063045 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.063012 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh"] Oct 30 21:47:44.067881 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.067863 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100497 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100465 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100653 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100505 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100653 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100530 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100756 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100655 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100756 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100736 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lxkl\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-kube-api-access-2lxkl\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100841 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100766 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100841 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100783 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273029a7-c256-495f-8a15-8b8a295e0885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100841 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100802 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.100955 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.100914 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273029a7-c256-495f-8a15-8b8a295e0885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.119192 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.119162 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dvk6k\"" Oct 30 21:47:44.165410 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.165383 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 21:47:44.168750 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.168724 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh"] Oct 30 21:47:44.201605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201570 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.201605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201602 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.201838 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201619 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.201838 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201647 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.201838 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201683 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lxkl\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-kube-api-access-2lxkl\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.201838 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201708 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.201838 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201736 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273029a7-c256-495f-8a15-8b8a295e0885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.201838 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201757 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.201838 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.201785 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273029a7-c256-495f-8a15-8b8a295e0885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.202191 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.202081 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.202191 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.202108 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.202191 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.202160 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.202298 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.202188 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.202512 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.202486 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273029a7-c256-495f-8a15-8b8a295e0885-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.204189 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.204160 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.204291 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.204274 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273029a7-c256-495f-8a15-8b8a295e0885-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.265005 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.264966 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.265141 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.265128 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lxkl\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-kube-api-access-2lxkl\") pod \"maas-default-gateway-openshift-default-7df854bf67-pwbsh\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.380519 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.380437 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:44.801092 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:44.801061 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh"] Oct 30 21:47:44.803943 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:47:44.803909 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273029a7_c256_495f_8a15_8b8a295e0885.slice/crio-cd97e5fd7b926f0899d7c0fe0241fb5090a2b7b3c48cf8be36ab8ed78ba3116e WatchSource:0}: Error finding container cd97e5fd7b926f0899d7c0fe0241fb5090a2b7b3c48cf8be36ab8ed78ba3116e: Status 404 returned error can't find the container with id cd97e5fd7b926f0899d7c0fe0241fb5090a2b7b3c48cf8be36ab8ed78ba3116e Oct 30 21:47:45.643752 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:45.643712 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" event={"ID":"273029a7-c256-495f-8a15-8b8a295e0885","Type":"ContainerStarted","Data":"cd97e5fd7b926f0899d7c0fe0241fb5090a2b7b3c48cf8be36ab8ed78ba3116e"} Oct 30 21:47:48.806518 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:48.806481 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:48.806910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:48.806567 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:48.806910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:48.806615 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:47:49.688590 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:49.688543 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" event={"ID":"273029a7-c256-495f-8a15-8b8a295e0885","Type":"ContainerStarted","Data":"5ca8f63063850253d74a219f9a42a021dc9392397d3b15002b4a8999c78aa422"} Oct 30 21:47:49.812635 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:47:49.812594 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:47:49.851328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:49.851261 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podStartSLOduration=1.851772814 podStartE2EDuration="5.851243659s" totalImagesPullingTime="3.999470845s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:49.850576515 +0000 UTC m=+578.441862013" watchObservedRunningTime="2026-10-30 21:47:49.851243659 +0000 UTC m=+578.442529146" Oct 30 21:47:50.384277 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:50.383477 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:50.384499 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:50.384456 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:47:50.384587 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:50.384514 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:47:51.381887 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:51.381848 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:47:51.382506 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:51.382471 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:47:52.383899 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:52.383856 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:47:52.384916 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:52.384488 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:47:53.381698 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:53.381649 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:47:53.382313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:53.382068 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:47:54.384573 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:54.384531 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:47:54.385304 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:54.385264 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:47:55.380987 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:55.380946 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:47:55.381164 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:55.381019 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:47:56.381835 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:56.381799 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 21:47:56.382267 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:56.381856 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 21:47:57.384597 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:57.384567 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:57.385088 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:57.384949 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:47:57.385719 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:47:57.385699 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 21:48:19.803400 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:48:19.803356 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:48:49.803559 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:48:49.803521 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:49:02.501452 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.501362 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf"] Oct 30 21:49:02.504508 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.504484 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.550958 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.550914 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-credential-socket\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.551143 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.550966 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gv5x\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-kube-api-access-7gv5x\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.551204 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.551135 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511034d-4cf2-4082-9613-e8896dfece70-istio-podinfo\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.551317 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.551297 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-socket\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.551382 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.551330 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-data\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.551628 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.551607 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-istio-token\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.551773 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.551671 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-certs\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.551773 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.551725 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-envoy\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.551883 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.551860 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511034d-4cf2-4082-9613-e8896dfece70-istiod-ca-cert\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.564798 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.564766 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ebf644d4-openshift-default-dockercfg-ws855\"" Oct 30 21:49:02.601593 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.601557 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf"] Oct 30 21:49:02.652334 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652299 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-credential-socket\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652334 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652334 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gv5x\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-kube-api-access-7gv5x\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652360 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511034d-4cf2-4082-9613-e8896dfece70-istio-podinfo\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652386 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-socket\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652408 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-data\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652463 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-istio-token\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652502 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-certs\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652554 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-envoy\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652611 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511034d-4cf2-4082-9613-e8896dfece70-istiod-ca-cert\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652958 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652857 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-credential-socket\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.652958 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652906 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-data\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.653036 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.652957 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-socket\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.653036 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.653015 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-certs\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.653260 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.653236 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511034d-4cf2-4082-9613-e8896dfece70-istiod-ca-cert\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.654905 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.654877 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-envoy\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.655153 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.655136 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511034d-4cf2-4082-9613-e8896dfece70-istio-podinfo\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.715927 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.715889 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gv5x\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-kube-api-access-7gv5x\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.716087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.715892 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-istio-token\") pod \"e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:02.817261 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:02.817171 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:03.242105 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:03.242073 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf"] Oct 30 21:49:03.248963 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:03.248918 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6511034d_4cf2_4082_9613_e8896dfece70.slice/crio-8f038c82ce09b6e83d36dd74d0f9f8ea5b28d34481224227f9bedc1e37b18d69 WatchSource:0}: Error finding container 8f038c82ce09b6e83d36dd74d0f9f8ea5b28d34481224227f9bedc1e37b18d69: Status 404 returned error can't find the container with id 8f038c82ce09b6e83d36dd74d0f9f8ea5b28d34481224227f9bedc1e37b18d69 Oct 30 21:49:03.251159 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:03.251125 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:03.251224 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:03.251197 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:03.251264 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:03.251227 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:04.001535 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.001498 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" event={"ID":"6511034d-4cf2-4082-9613-e8896dfece70","Type":"ContainerStarted","Data":"8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998"} Oct 30 21:49:04.001980 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.001558 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" event={"ID":"6511034d-4cf2-4082-9613-e8896dfece70","Type":"ContainerStarted","Data":"8f038c82ce09b6e83d36dd74d0f9f8ea5b28d34481224227f9bedc1e37b18d69"} Oct 30 21:49:04.162649 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.162592 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" podStartSLOduration=2.162576622 podStartE2EDuration="2.162576622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:04.161084657 +0000 UTC m=+652.752370145" watchObservedRunningTime="2026-10-30 21:49:04.162576622 +0000 UTC m=+652.753862109" Oct 30 21:49:04.782828 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.782792 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk"] Oct 30 21:49:04.785830 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.785801 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.817981 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.817950 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:04.823540 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.823510 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:04.836196 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.836171 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-09d8fb9b-openshift-default-dockercfg-n6j22\"" Oct 30 21:49:04.870737 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.870707 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-certs\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.870894 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.870748 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-credential-socket\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.870894 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.870780 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-socket\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.870976 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.870911 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istiod-ca-cert\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.870976 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.870960 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-data\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.871064 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.871038 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-podinfo\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.871186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.871092 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-token\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.871239 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.871222 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8r7bf\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-kube-api-access-8r7bf\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.871282 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.871265 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-envoy\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.884109 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.884077 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk"] Oct 30 21:49:04.972629 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972593 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-certs\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.972629 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972634 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-credential-socket\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.972890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972656 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-socket\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.972890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972703 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istiod-ca-cert\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.972890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972727 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-data\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.972890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972753 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-podinfo\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.972890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972793 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-token\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.972890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972849 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8r7bf\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-kube-api-access-8r7bf\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.972890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.972891 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-envoy\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.973175 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.973063 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-certs\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.973368 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.973342 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-credential-socket\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.973513 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.973464 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-socket\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.973513 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.973490 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istiod-ca-cert\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.973630 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.973539 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-data\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.975360 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.975341 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-podinfo\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:04.975454 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:04.975357 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-envoy\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:05.006243 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.006218 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:05.007111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.007096 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:05.033355 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.033287 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-token\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:05.037696 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.037667 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8r7bf\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-kube-api-access-8r7bf\") pod \"e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:05.103644 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.103617 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:05.523555 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.523525 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk"] Oct 30 21:49:05.529935 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:05.529905 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da58100_bf31_4805_94dd_c5a34dfd3ff9.slice/crio-f09e2899c68b6dc36bd44a86118db6bdba45452afaa76c755dd61bf9f6fd5751 WatchSource:0}: Error finding container f09e2899c68b6dc36bd44a86118db6bdba45452afaa76c755dd61bf9f6fd5751: Status 404 returned error can't find the container with id f09e2899c68b6dc36bd44a86118db6bdba45452afaa76c755dd61bf9f6fd5751 Oct 30 21:49:05.532209 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.532177 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:05.532274 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.532240 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:05.532313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:05.532271 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:06.012270 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:06.012236 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" event={"ID":"0da58100-bf31-4805-94dd-c5a34dfd3ff9","Type":"ContainerStarted","Data":"adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467"} Oct 30 21:49:06.012652 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:06.012298 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" event={"ID":"0da58100-bf31-4805-94dd-c5a34dfd3ff9","Type":"ContainerStarted","Data":"f09e2899c68b6dc36bd44a86118db6bdba45452afaa76c755dd61bf9f6fd5751"} Oct 30 21:49:06.104653 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:06.104623 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:06.109952 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:06.109928 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:06.134189 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:06.134136 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" podStartSLOduration=2.1341210999999998 podStartE2EDuration="2.1341211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:06.132100485 +0000 UTC m=+654.723385972" watchObservedRunningTime="2026-10-30 21:49:06.1341211 +0000 UTC m=+654.725406587" Oct 30 21:49:07.018555 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:07.018519 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:07.019633 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:07.019611 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:09.253431 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.253392 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc"] Oct 30 21:49:09.256227 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.256208 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:09.309694 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.309665 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a1d107b0-dockercfg-hsscr\"" Oct 30 21:49:09.316121 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.316089 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzhgn\" (UniqueName: \"kubernetes.io/projected/b1392c96-2823-4563-97ab-cadb52cf94f9-kube-api-access-zzhgn\") pod \"payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc\" (UID: \"b1392c96-2823-4563-97ab-cadb52cf94f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:09.316288 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.316262 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1392c96-2823-4563-97ab-cadb52cf94f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc\" (UID: \"b1392c96-2823-4563-97ab-cadb52cf94f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:09.356762 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.356730 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a1d107b0\"" Oct 30 21:49:09.358212 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.358190 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc"] Oct 30 21:49:09.386167 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.386138 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv"] Oct 30 21:49:09.388766 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.388743 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:09.417196 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.417167 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b9354c-5b48-4154-b77d-c30adb5ad520-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv\" (UID: \"67b9354c-5b48-4154-b77d-c30adb5ad520\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:09.417318 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.417209 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1392c96-2823-4563-97ab-cadb52cf94f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc\" (UID: \"b1392c96-2823-4563-97ab-cadb52cf94f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:09.417318 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.417300 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm4p9\" (UniqueName: \"kubernetes.io/projected/67b9354c-5b48-4154-b77d-c30adb5ad520-kube-api-access-sm4p9\") pod \"payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv\" (UID: \"67b9354c-5b48-4154-b77d-c30adb5ad520\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:09.417462 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.417363 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzhgn\" (UniqueName: \"kubernetes.io/projected/b1392c96-2823-4563-97ab-cadb52cf94f9-kube-api-access-zzhgn\") pod \"payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc\" (UID: \"b1392c96-2823-4563-97ab-cadb52cf94f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:09.418045 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.418023 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1392c96-2823-4563-97ab-cadb52cf94f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc\" (UID: \"b1392c96-2823-4563-97ab-cadb52cf94f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:09.449697 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.449662 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv"] Oct 30 21:49:09.475660 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.475631 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzhgn\" (UniqueName: \"kubernetes.io/projected/b1392c96-2823-4563-97ab-cadb52cf94f9-kube-api-access-zzhgn\") pod \"payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc\" (UID: \"b1392c96-2823-4563-97ab-cadb52cf94f9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:09.518126 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.518037 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm4p9\" (UniqueName: \"kubernetes.io/projected/67b9354c-5b48-4154-b77d-c30adb5ad520-kube-api-access-sm4p9\") pod \"payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv\" (UID: \"67b9354c-5b48-4154-b77d-c30adb5ad520\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:09.518286 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.518164 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b9354c-5b48-4154-b77d-c30adb5ad520-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv\" (UID: \"67b9354c-5b48-4154-b77d-c30adb5ad520\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:09.518902 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.518881 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b9354c-5b48-4154-b77d-c30adb5ad520-plugins-config-volume\") pod \"payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv\" (UID: \"67b9354c-5b48-4154-b77d-c30adb5ad520\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:09.568594 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.568557 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:09.576186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.576152 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm4p9\" (UniqueName: \"kubernetes.io/projected/67b9354c-5b48-4154-b77d-c30adb5ad520-kube-api-access-sm4p9\") pod \"payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv\" (UID: \"67b9354c-5b48-4154-b77d-c30adb5ad520\") " pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:09.700934 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.700897 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:09.963890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:09.963815 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc"] Oct 30 21:49:09.969778 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:09.969751 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1392c96_2823_4563_97ab_cadb52cf94f9.slice/crio-edb6e0ea2fb8ff79bd1d600eaf7ed53bcfddcd2d1c076b7ce5ac92e0edaeee25 WatchSource:0}: Error finding container edb6e0ea2fb8ff79bd1d600eaf7ed53bcfddcd2d1c076b7ce5ac92e0edaeee25: Status 404 returned error can't find the container with id edb6e0ea2fb8ff79bd1d600eaf7ed53bcfddcd2d1c076b7ce5ac92e0edaeee25 Oct 30 21:49:10.031965 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:10.031932 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" event={"ID":"b1392c96-2823-4563-97ab-cadb52cf94f9","Type":"ContainerStarted","Data":"5e251ffe4ac396be6d821e521a2b85df6d9fbb26aff38a94ed846cafedfdd851"} Oct 30 21:49:10.032061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:10.031980 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" event={"ID":"b1392c96-2823-4563-97ab-cadb52cf94f9","Type":"ContainerStarted","Data":"edb6e0ea2fb8ff79bd1d600eaf7ed53bcfddcd2d1c076b7ce5ac92e0edaeee25"} Oct 30 21:49:10.032103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:10.032068 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:10.074506 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:10.074477 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv"] Oct 30 21:49:10.080970 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:10.080946 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b9354c_5b48_4154_b77d_c30adb5ad520.slice/crio-ccfb6ebb48ec5c91805b396ede4f04e098b980d96a6d108e0030dbce7000d8c4 WatchSource:0}: Error finding container ccfb6ebb48ec5c91805b396ede4f04e098b980d96a6d108e0030dbce7000d8c4: Status 404 returned error can't find the container with id ccfb6ebb48ec5c91805b396ede4f04e098b980d96a6d108e0030dbce7000d8c4 Oct 30 21:49:10.146118 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:10.146070 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" podStartSLOduration=1.146055138 podStartE2EDuration="1.146055138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:10.145479182 +0000 UTC m=+658.736764670" watchObservedRunningTime="2026-10-30 21:49:10.146055138 +0000 UTC m=+658.737340632" Oct 30 21:49:11.039886 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.039846 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" event={"ID":"67b9354c-5b48-4154-b77d-c30adb5ad520","Type":"ContainerStarted","Data":"98b3a7b3f05f9e146b75a7b03e91fcd1a85bfc40339292999b2954c5e1c8a435"} Oct 30 21:49:11.040274 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.039907 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" event={"ID":"67b9354c-5b48-4154-b77d-c30adb5ad520","Type":"ContainerStarted","Data":"ccfb6ebb48ec5c91805b396ede4f04e098b980d96a6d108e0030dbce7000d8c4"} Oct 30 21:49:11.040274 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.040018 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:11.150956 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.150772 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" podStartSLOduration=2.150758846 podStartE2EDuration="2.150758846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:11.150477075 +0000 UTC m=+659.741762563" watchObservedRunningTime="2026-10-30 21:49:11.150758846 +0000 UTC m=+659.742044362" Oct 30 21:49:11.592780 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.592740 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w"] Oct 30 21:49:11.595895 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.595867 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.640478 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.640444 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.640641 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.640529 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386a0ef0-71a7-4aa0-a18d-d46318876bc7-openshift-service-ca\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.640641 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.640558 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-metrics-certs\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.640641 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.640587 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57jb\" (UniqueName: \"kubernetes.io/projected/386a0ef0-71a7-4aa0-a18d-d46318876bc7-kube-api-access-v57jb\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.688454 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.688403 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w"] Oct 30 21:49:11.692451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.690726 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ebf644d4-serving-cert\"" Oct 30 21:49:11.741475 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.741443 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386a0ef0-71a7-4aa0-a18d-d46318876bc7-openshift-service-ca\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.741681 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.741483 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-metrics-certs\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.741681 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.741519 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57jb\" (UniqueName: \"kubernetes.io/projected/386a0ef0-71a7-4aa0-a18d-d46318876bc7-kube-api-access-v57jb\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.741681 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.741582 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.741851 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:11.741686 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-serving-cert: secret "maas-api-e2e-ait-ebf644d4-serving-cert" not found Oct 30 21:49:11.741908 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:11.741876 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls podName:386a0ef0-71a7-4aa0-a18d-d46318876bc7 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:12.241838245 +0000 UTC m=+660.833123736 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls") pod "maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" (UID: "386a0ef0-71a7-4aa0-a18d-d46318876bc7") : secret "maas-api-e2e-ait-ebf644d4-serving-cert" not found Oct 30 21:49:11.742175 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.742154 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386a0ef0-71a7-4aa0-a18d-d46318876bc7-openshift-service-ca\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.743976 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.743951 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-metrics-certs\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:11.799260 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:11.799226 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57jb\" (UniqueName: \"kubernetes.io/projected/386a0ef0-71a7-4aa0-a18d-d46318876bc7-kube-api-access-v57jb\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:12.246768 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:12.246731 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:12.249178 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:12.249149 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls\") pod \"maas-api-e2e-ait-ebf644d4-59cf567954-tps4w\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:12.510065 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:12.509967 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:12.883008 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:12.882922 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w"] Oct 30 21:49:12.889239 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:12.889204 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386a0ef0_71a7_4aa0_a18d_d46318876bc7.slice/crio-943e1c4666b54851e25b344bd122f55c775a6f56198617002798a94b186d8095 WatchSource:0}: Error finding container 943e1c4666b54851e25b344bd122f55c775a6f56198617002798a94b186d8095: Status 404 returned error can't find the container with id 943e1c4666b54851e25b344bd122f55c775a6f56198617002798a94b186d8095 Oct 30 21:49:13.051094 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:13.051059 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" event={"ID":"386a0ef0-71a7-4aa0-a18d-d46318876bc7","Type":"ContainerStarted","Data":"943e1c4666b54851e25b344bd122f55c775a6f56198617002798a94b186d8095"} Oct 30 21:49:14.056513 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.056476 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" event={"ID":"386a0ef0-71a7-4aa0-a18d-d46318876bc7","Type":"ContainerStarted","Data":"a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6"} Oct 30 21:49:14.056915 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.056539 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:14.170678 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.170565 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" podStartSLOduration=2.176502981 podStartE2EDuration="3.170552651s" totalImagesPullingTime="994.04967ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:14.169508638 +0000 UTC m=+662.760794122" watchObservedRunningTime="2026-10-30 21:49:14.170552651 +0000 UTC m=+662.761838138" Oct 30 21:49:14.407373 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.407341 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k"] Oct 30 21:49:14.410045 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.410027 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:14.459589 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.459530 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ebf644d4\"" Oct 30 21:49:14.459589 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.459555 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ebf644d4-dockercfg-6pgct\"" Oct 30 21:49:14.479644 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.479604 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed05a9a-2235-4053-ae44-6049f835490e-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k\" (UID: \"eed05a9a-2235-4053-ae44-6049f835490e\") " pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:14.479799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.479734 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkxk\" (UniqueName: \"kubernetes.io/projected/eed05a9a-2235-4053-ae44-6049f835490e-kube-api-access-7fkxk\") pod \"payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k\" (UID: \"eed05a9a-2235-4053-ae44-6049f835490e\") " pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:14.482807 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.482781 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k"] Oct 30 21:49:14.580524 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.580493 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed05a9a-2235-4053-ae44-6049f835490e-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k\" (UID: \"eed05a9a-2235-4053-ae44-6049f835490e\") " pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:14.580709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.580568 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkxk\" (UniqueName: \"kubernetes.io/projected/eed05a9a-2235-4053-ae44-6049f835490e-kube-api-access-7fkxk\") pod \"payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k\" (UID: \"eed05a9a-2235-4053-ae44-6049f835490e\") " pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:14.581222 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.581201 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed05a9a-2235-4053-ae44-6049f835490e-plugins-config-volume\") pod \"payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k\" (UID: \"eed05a9a-2235-4053-ae44-6049f835490e\") " pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:14.638981 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.638947 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkxk\" (UniqueName: \"kubernetes.io/projected/eed05a9a-2235-4053-ae44-6049f835490e-kube-api-access-7fkxk\") pod \"payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k\" (UID: \"eed05a9a-2235-4053-ae44-6049f835490e\") " pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:14.721988 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:14.721907 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:15.087784 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:15.087751 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k"] Oct 30 21:49:15.093779 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:15.093749 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed05a9a_2235_4053_ae44_6049f835490e.slice/crio-b976595cb7eda7fcaeb46d6d925915bf75d9b75f681eeb6d560f920e4cb04487 WatchSource:0}: Error finding container b976595cb7eda7fcaeb46d6d925915bf75d9b75f681eeb6d560f920e4cb04487: Status 404 returned error can't find the container with id b976595cb7eda7fcaeb46d6d925915bf75d9b75f681eeb6d560f920e4cb04487 Oct 30 21:49:16.067337 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.067301 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" event={"ID":"eed05a9a-2235-4053-ae44-6049f835490e","Type":"ContainerStarted","Data":"7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17"} Oct 30 21:49:16.067544 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.067359 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" event={"ID":"eed05a9a-2235-4053-ae44-6049f835490e","Type":"ContainerStarted","Data":"b976595cb7eda7fcaeb46d6d925915bf75d9b75f681eeb6d560f920e4cb04487"} Oct 30 21:49:16.067544 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.067408 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:16.177046 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.176987 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" podStartSLOduration=2.176972265 podStartE2EDuration="2.176972265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:16.175663839 +0000 UTC m=+664.766949326" watchObservedRunningTime="2026-10-30 21:49:16.176972265 +0000 UTC m=+664.768257752" Oct 30 21:49:16.637246 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.637213 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45"] Oct 30 21:49:16.640000 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.639981 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.689057 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.689014 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-09d8fb9b-serving-cert\"" Oct 30 21:49:16.701190 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.701157 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45"] Oct 30 21:49:16.701658 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.701622 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.701658 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.701651 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaade99c-d02b-401c-b71f-77bb257cb7da-openshift-service-ca\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.701823 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.701794 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-metrics-certs\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.701865 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.701841 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42jcw\" (UniqueName: \"kubernetes.io/projected/aaade99c-d02b-401c-b71f-77bb257cb7da-kube-api-access-42jcw\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.802840 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.802806 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.802840 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.802843 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaade99c-d02b-401c-b71f-77bb257cb7da-openshift-service-ca\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.803059 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.802878 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-metrics-certs\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.803059 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.802901 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42jcw\" (UniqueName: \"kubernetes.io/projected/aaade99c-d02b-401c-b71f-77bb257cb7da-kube-api-access-42jcw\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.803059 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:16.802965 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-serving-cert: secret "maas-api-e2e-ait-09d8fb9b-serving-cert" not found Oct 30 21:49:16.803059 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:16.803051 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls podName:aaade99c-d02b-401c-b71f-77bb257cb7da nodeName:}" failed. No retries permitted until 2026-10-30 21:49:17.303033439 +0000 UTC m=+665.894318904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls") pod "maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" (UID: "aaade99c-d02b-401c-b71f-77bb257cb7da") : secret "maas-api-e2e-ait-09d8fb9b-serving-cert" not found Oct 30 21:49:16.803536 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.803514 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaade99c-d02b-401c-b71f-77bb257cb7da-openshift-service-ca\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.805348 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.805332 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-metrics-certs\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:16.865807 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:16.865777 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42jcw\" (UniqueName: \"kubernetes.io/projected/aaade99c-d02b-401c-b71f-77bb257cb7da-kube-api-access-42jcw\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:17.309049 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:17.309009 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:17.311486 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:17.311466 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls\") pod \"maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:17.551525 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:17.551491 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:17.920872 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:17.920787 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45"] Oct 30 21:49:17.927000 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:17.926967 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaade99c_d02b_401c_b71f_77bb257cb7da.slice/crio-81aaa4de201ed75a9e7d626bbb53b5df32d39942116c40bd7e772d6bd8ff0a5a WatchSource:0}: Error finding container 81aaa4de201ed75a9e7d626bbb53b5df32d39942116c40bd7e772d6bd8ff0a5a: Status 404 returned error can't find the container with id 81aaa4de201ed75a9e7d626bbb53b5df32d39942116c40bd7e772d6bd8ff0a5a Oct 30 21:49:18.081730 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:18.081694 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" event={"ID":"aaade99c-d02b-401c-b71f-77bb257cb7da","Type":"ContainerStarted","Data":"81aaa4de201ed75a9e7d626bbb53b5df32d39942116c40bd7e772d6bd8ff0a5a"} Oct 30 21:49:19.088348 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:19.088246 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" event={"ID":"aaade99c-d02b-401c-b71f-77bb257cb7da","Type":"ContainerStarted","Data":"d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6"} Oct 30 21:49:19.088741 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:19.088356 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:19.201068 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:19.201008 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" podStartSLOduration=2.3181058119999998 podStartE2EDuration="3.200990409s" totalImagesPullingTime="882.884597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:19.200318407 +0000 UTC m=+667.791603896" watchObservedRunningTime="2026-10-30 21:49:19.200990409 +0000 UTC m=+667.792275893" Oct 30 21:49:19.803823 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:19.803781 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:49:20.067989 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:20.067922 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:49:21.043490 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.043461 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:49:21.766589 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.766554 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv"] Oct 30 21:49:21.769961 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.769939 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.818907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.818871 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a1d107b0-serving-cert\"" Oct 30 21:49:21.856382 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.856344 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-metrics-certs\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.856547 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.856508 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0a5e91-3fde-415a-a14d-5f128365afd4-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.856593 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.856579 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.856666 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.856640 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbsqt\" (UniqueName: \"kubernetes.io/projected/ce0a5e91-3fde-415a-a14d-5f128365afd4-kube-api-access-hbsqt\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.866873 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.866849 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv"] Oct 30 21:49:21.958093 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.958058 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-metrics-certs\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.958261 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.958133 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0a5e91-3fde-415a-a14d-5f128365afd4-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.958261 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.958205 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.958261 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.958227 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbsqt\" (UniqueName: \"kubernetes.io/projected/ce0a5e91-3fde-415a-a14d-5f128365afd4-kube-api-access-hbsqt\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.958381 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:21.958356 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-serving-cert: secret "maas-api-e2e-ait-a1d107b0-serving-cert" not found Oct 30 21:49:21.958496 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:21.958482 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls podName:ce0a5e91-3fde-415a-a14d-5f128365afd4 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:22.458458576 +0000 UTC m=+671.049744050 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls") pod "maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" (UID: "ce0a5e91-3fde-415a-a14d-5f128365afd4") : secret "maas-api-e2e-ait-a1d107b0-serving-cert" not found Oct 30 21:49:21.958966 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.958949 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0a5e91-3fde-415a-a14d-5f128365afd4-openshift-service-ca\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:21.960476 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:21.960454 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-metrics-certs\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:22.022666 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:22.022597 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbsqt\" (UniqueName: \"kubernetes.io/projected/ce0a5e91-3fde-415a-a14d-5f128365afd4-kube-api-access-hbsqt\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:22.047970 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:22.047941 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:49:22.462639 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:22.462559 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:22.464931 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:22.464894 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls\") pod \"maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:22.683365 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:22.683328 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:23.105484 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:23.105452 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv"] Oct 30 21:49:23.112126 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:23.112098 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0a5e91_3fde_415a_a14d_5f128365afd4.slice/crio-7be614f312179a5042929437d3a12d323642ab541ea0c9b5c2da98a837b76e03 WatchSource:0}: Error finding container 7be614f312179a5042929437d3a12d323642ab541ea0c9b5c2da98a837b76e03: Status 404 returned error can't find the container with id 7be614f312179a5042929437d3a12d323642ab541ea0c9b5c2da98a837b76e03 Oct 30 21:49:24.111956 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:24.111919 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" event={"ID":"ce0a5e91-3fde-415a-a14d-5f128365afd4","Type":"ContainerStarted","Data":"4711040cbb3dcc37920b4cd71277bcd3e2f01cfece7ddcacb78048f3ad74f96a"} Oct 30 21:49:24.112317 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:24.111973 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" event={"ID":"ce0a5e91-3fde-415a-a14d-5f128365afd4","Type":"ContainerStarted","Data":"7be614f312179a5042929437d3a12d323642ab541ea0c9b5c2da98a837b76e03"} Oct 30 21:49:24.112317 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:24.112054 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:24.224622 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:24.224563 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" podStartSLOduration=2.338620324 podStartE2EDuration="3.224546529s" totalImagesPullingTime="885.926205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:24.223441879 +0000 UTC m=+672.814727357" watchObservedRunningTime="2026-10-30 21:49:24.224546529 +0000 UTC m=+672.815832015" Oct 30 21:49:25.102612 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:25.102567 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:49:27.112670 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:27.112634 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:49:30.126543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:30.126515 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:49:31.254310 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.254269 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg"] Oct 30 21:49:31.259677 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.259650 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.291988 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.291951 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.292155 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.292018 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-metrics-certs\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.292217 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.292144 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ddc82-b923-40c2-8d90-c172f0494fe1-openshift-service-ca\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.292304 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.292288 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r5rs\" (UniqueName: \"kubernetes.io/projected/9d9ddc82-b923-40c2-8d90-c172f0494fe1-kube-api-access-5r5rs\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.356008 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.355943 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-16c49820-serving-cert\"" Oct 30 21:49:31.358610 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.358577 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg"] Oct 30 21:49:31.393884 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.393842 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ddc82-b923-40c2-8d90-c172f0494fe1-openshift-service-ca\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.394127 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.393937 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r5rs\" (UniqueName: \"kubernetes.io/projected/9d9ddc82-b923-40c2-8d90-c172f0494fe1-kube-api-access-5r5rs\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.394127 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.394011 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.394127 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.394052 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-metrics-certs\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.394969 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:31.394945 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-serving-cert: secret "maas-api-e2e-migrate-16c49820-serving-cert" not found Oct 30 21:49:31.395187 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:31.395158 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls podName:9d9ddc82-b923-40c2-8d90-c172f0494fe1 nodeName:}" failed. No retries permitted until 2026-10-30 21:49:31.895136285 +0000 UTC m=+680.486421764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls") pod "maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" (UID: "9d9ddc82-b923-40c2-8d90-c172f0494fe1") : secret "maas-api-e2e-migrate-16c49820-serving-cert" not found Oct 30 21:49:31.395319 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.395190 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ddc82-b923-40c2-8d90-c172f0494fe1-openshift-service-ca\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.404975 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.404945 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-metrics-certs\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.452989 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.452955 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r5rs\" (UniqueName: \"kubernetes.io/projected/9d9ddc82-b923-40c2-8d90-c172f0494fe1-kube-api-access-5r5rs\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.899885 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.899848 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:31.902496 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:31.902458 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls\") pod \"maas-api-e2e-migrate-16c49820-74bc47897c-57mbg\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:32.176087 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:32.175991 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:32.553597 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:32.553559 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg"] Oct 30 21:49:32.560616 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:49:32.560576 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d9ddc82_b923_40c2_8d90_c172f0494fe1.slice/crio-8794870cbf4453efe3b040e5547fa08f7ddca332fee7fd5daedbb11bdc7d2321 WatchSource:0}: Error finding container 8794870cbf4453efe3b040e5547fa08f7ddca332fee7fd5daedbb11bdc7d2321: Status 404 returned error can't find the container with id 8794870cbf4453efe3b040e5547fa08f7ddca332fee7fd5daedbb11bdc7d2321 Oct 30 21:49:33.161199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:33.161158 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" event={"ID":"9d9ddc82-b923-40c2-8d90-c172f0494fe1","Type":"ContainerStarted","Data":"8794870cbf4453efe3b040e5547fa08f7ddca332fee7fd5daedbb11bdc7d2321"} Oct 30 21:49:34.167398 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.167356 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" event={"ID":"9d9ddc82-b923-40c2-8d90-c172f0494fe1","Type":"ContainerStarted","Data":"8eaf7c8ecf9eda73ed9b95134001d5d68e98c5416350a0dfa389125e4fbcd8b1"} Oct 30 21:49:34.167940 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.167439 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:34.168925 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.168899 5139 generic.go:356] "Generic (PLEG): container finished" podID="6511034d-4cf2-4082-9613-e8896dfece70" containerID="8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998" exitCode=0 Oct 30 21:49:34.169071 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.168981 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" event={"ID":"6511034d-4cf2-4082-9613-e8896dfece70","Type":"ContainerDied","Data":"8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998"} Oct 30 21:49:34.169366 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.169342 5139 scope.go:122] "RemoveContainer" containerID="8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998" Oct 30 21:49:34.170470 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.170448 5139 generic.go:356] "Generic (PLEG): container finished" podID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerID="adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467" exitCode=0 Oct 30 21:49:34.170546 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.170499 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" event={"ID":"0da58100-bf31-4805-94dd-c5a34dfd3ff9","Type":"ContainerDied","Data":"adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467"} Oct 30 21:49:34.170617 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.170590 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.170670 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.170657 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.170713 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.170693 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.170799 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.170787 5139 scope.go:122] "RemoveContainer" containerID="adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467" Oct 30 21:49:34.171835 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.171809 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.171899 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.171869 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.171935 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.171902 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:49:34.285154 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:34.285090 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" podStartSLOduration=2.36604989 podStartE2EDuration="3.285071644s" totalImagesPullingTime="919.021754ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:34.28198089 +0000 UTC m=+682.873266402" watchObservedRunningTime="2026-10-30 21:49:34.285071644 +0000 UTC m=+682.876357137" Oct 30 21:49:35.008556 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:35.008522 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:35.217459 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:35.217320 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" event={"ID":"6511034d-4cf2-4082-9613-e8896dfece70","Type":"ContainerStarted","Data":"15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2"} Oct 30 21:49:35.221205 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:35.221166 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" event={"ID":"0da58100-bf31-4805-94dd-c5a34dfd3ff9","Type":"ContainerStarted","Data":"24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30"} Oct 30 21:49:35.818520 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:35.818487 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:35.819153 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:35.819128 5139 patch_prober.go:29] interesting pod/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:49:35.819290 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:35.819176 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:49:36.107075 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:36.106982 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:36.107439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:36.107397 5139 patch_prober.go:29] interesting pod/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:49:36.107612 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:36.107586 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:49:36.817757 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:36.817712 5139 patch_prober.go:29] interesting pod/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:49:36.818271 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:36.817780 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:49:37.018715 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:37.018674 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:37.107501 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:37.106804 5139 patch_prober.go:29] interesting pod/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:49:37.107501 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:37.106871 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:49:37.817917 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:37.817883 5139 patch_prober.go:29] interesting pod/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:49:37.818375 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:37.818349 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:49:38.106923 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:38.106821 5139 patch_prober.go:29] interesting pod/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:49:38.107214 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:38.107180 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:49:38.817673 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:38.817636 5139 patch_prober.go:29] interesting pod/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:49:38.817881 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:38.817702 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:49:39.111497 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:39.110817 5139 patch_prober.go:29] interesting pod/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:49:39.111497 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:39.110883 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:49:39.818201 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:39.818159 5139 patch_prober.go:29] interesting pod/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 30 21:49:39.818502 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:39.818469 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 30 21:49:40.110727 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:40.110631 5139 patch_prober.go:29] interesting pod/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 30 21:49:40.111002 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:40.110975 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 30 21:49:40.226672 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:40.226634 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:49:40.822489 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:40.822453 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:40.822750 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:40.822730 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:40.823454 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:40.823439 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:49:41.108624 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:41.108544 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:41.109774 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:41.109752 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:49:49.809760 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:49:49.809715 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:49:58.391238 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:49:58.391195 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k"] Oct 30 21:50:03.391904 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:03.391839 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" podUID="eed05a9a-2235-4053-ae44-6049f835490e" containerName="payload-processing" containerID="cri-o://7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17" gracePeriod=25 Oct 30 21:50:03.649828 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:03.649763 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:50:03.754928 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:03.754886 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed05a9a-2235-4053-ae44-6049f835490e-kube-api-access-7fkxk\" (UniqueName: \"kubernetes.io/projected/eed05a9a-2235-4053-ae44-6049f835490e-kube-api-access-7fkxk\") pod \"eed05a9a-2235-4053-ae44-6049f835490e\" (UID: \"eed05a9a-2235-4053-ae44-6049f835490e\") " Oct 30 21:50:03.755103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:03.755025 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed05a9a-2235-4053-ae44-6049f835490e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed05a9a-2235-4053-ae44-6049f835490e-plugins-config-volume\") pod \"eed05a9a-2235-4053-ae44-6049f835490e\" (UID: \"eed05a9a-2235-4053-ae44-6049f835490e\") " Oct 30 21:50:03.755486 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:03.755465 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed05a9a-2235-4053-ae44-6049f835490e-plugins-config-volume" pod "eed05a9a-2235-4053-ae44-6049f835490e" (UID: "eed05a9a-2235-4053-ae44-6049f835490e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:03.757050 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:03.757023 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed05a9a-2235-4053-ae44-6049f835490e-kube-api-access-7fkxk" pod "eed05a9a-2235-4053-ae44-6049f835490e" (UID: "eed05a9a-2235-4053-ae44-6049f835490e"). InnerVolumeSpecName "kube-api-access-7fkxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:03.856519 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:03.856482 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fkxk\" (UniqueName: \"kubernetes.io/projected/eed05a9a-2235-4053-ae44-6049f835490e-kube-api-access-7fkxk\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:03.856519 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:03.856511 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eed05a9a-2235-4053-ae44-6049f835490e-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:04.354617 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.354582 5139 generic.go:356] "Generic (PLEG): container finished" podID="eed05a9a-2235-4053-ae44-6049f835490e" containerID="7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17" exitCode=0 Oct 30 21:50:04.354806 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.354629 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" Oct 30 21:50:04.354806 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.354665 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" event={"ID":"eed05a9a-2235-4053-ae44-6049f835490e","Type":"ContainerDied","Data":"7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17"} Oct 30 21:50:04.354806 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.354718 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k" event={"ID":"eed05a9a-2235-4053-ae44-6049f835490e","Type":"ContainerDied","Data":"b976595cb7eda7fcaeb46d6d925915bf75d9b75f681eeb6d560f920e4cb04487"} Oct 30 21:50:04.354806 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.354742 5139 scope.go:122] "RemoveContainer" containerID="7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17" Oct 30 21:50:04.363873 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.363854 5139 scope.go:122] "RemoveContainer" containerID="7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17" Oct 30 21:50:04.364229 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:04.364202 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17\": container with ID starting with 7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17 not found: ID does not exist" containerID="7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17" Oct 30 21:50:04.364311 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.364239 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17"} err="failed to get container status \"7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17\": rpc error: code = NotFound desc = could not find container \"7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17\": container with ID starting with 7f740e04b2daf6e494455cb8afc58ca54e365b0eb20c58473f760f6c3cc4fe17 not found: ID does not exist" Oct 30 21:50:04.508752 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.508715 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k"] Oct 30 21:50:04.555197 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:04.555158 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ebf644d4-66bffcbb8f-lqh8k"] Oct 30 21:50:05.541525 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:05.541491 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed05a9a-2235-4053-ae44-6049f835490e" path="/var/lib/kubelet/pods/eed05a9a-2235-4053-ae44-6049f835490e/volumes" Oct 30 21:50:09.464000 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.463951 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w"] Oct 30 21:50:09.464706 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.464210 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" podUID="386a0ef0-71a7-4aa0-a18d-d46318876bc7" containerName="maas-api" containerID="cri-o://a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6" gracePeriod=30 Oct 30 21:50:09.718879 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.718814 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:50:09.820696 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.820645 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/386a0ef0-71a7-4aa0-a18d-d46318876bc7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386a0ef0-71a7-4aa0-a18d-d46318876bc7-openshift-service-ca\") pod \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " Oct 30 21:50:09.820889 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.820759 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls\") pod \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " Oct 30 21:50:09.820935 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.820894 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/386a0ef0-71a7-4aa0-a18d-d46318876bc7-kube-api-access-v57jb\" (UniqueName: \"kubernetes.io/projected/386a0ef0-71a7-4aa0-a18d-d46318876bc7-kube-api-access-v57jb\") pod \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " Oct 30 21:50:09.820976 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.820942 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-metrics-certs\") pod \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\" (UID: \"386a0ef0-71a7-4aa0-a18d-d46318876bc7\") " Oct 30 21:50:09.821110 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.821086 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/386a0ef0-71a7-4aa0-a18d-d46318876bc7-openshift-service-ca" pod "386a0ef0-71a7-4aa0-a18d-d46318876bc7" (UID: "386a0ef0-71a7-4aa0-a18d-d46318876bc7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:09.821266 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.821246 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/386a0ef0-71a7-4aa0-a18d-d46318876bc7-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:09.822866 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.822838 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls" pod "386a0ef0-71a7-4aa0-a18d-d46318876bc7" (UID: "386a0ef0-71a7-4aa0-a18d-d46318876bc7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:09.822988 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.822916 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/386a0ef0-71a7-4aa0-a18d-d46318876bc7-kube-api-access-v57jb" pod "386a0ef0-71a7-4aa0-a18d-d46318876bc7" (UID: "386a0ef0-71a7-4aa0-a18d-d46318876bc7"). InnerVolumeSpecName "kube-api-access-v57jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:09.823033 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.823002 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-metrics-certs" pod "386a0ef0-71a7-4aa0-a18d-d46318876bc7" (UID: "386a0ef0-71a7-4aa0-a18d-d46318876bc7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:09.921920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.921888 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:09.921920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.921918 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/386a0ef0-71a7-4aa0-a18d-d46318876bc7-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:09.921920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:09.921929 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57jb\" (UniqueName: \"kubernetes.io/projected/386a0ef0-71a7-4aa0-a18d-d46318876bc7-kube-api-access-v57jb\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:10.387153 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.387117 5139 generic.go:356] "Generic (PLEG): container finished" podID="386a0ef0-71a7-4aa0-a18d-d46318876bc7" containerID="a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6" exitCode=0 Oct 30 21:50:10.387329 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.387171 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" Oct 30 21:50:10.387329 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.387203 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" event={"ID":"386a0ef0-71a7-4aa0-a18d-d46318876bc7","Type":"ContainerDied","Data":"a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6"} Oct 30 21:50:10.387329 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.387263 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w" event={"ID":"386a0ef0-71a7-4aa0-a18d-d46318876bc7","Type":"ContainerDied","Data":"943e1c4666b54851e25b344bd122f55c775a6f56198617002798a94b186d8095"} Oct 30 21:50:10.387329 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.387296 5139 scope.go:122] "RemoveContainer" containerID="a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6" Oct 30 21:50:10.398285 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.398260 5139 scope.go:122] "RemoveContainer" containerID="a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6" Oct 30 21:50:10.398652 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:10.398628 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6\": container with ID starting with a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6 not found: ID does not exist" containerID="a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6" Oct 30 21:50:10.398752 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.398664 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6"} err="failed to get container status \"a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6\": rpc error: code = NotFound desc = could not find container \"a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6\": container with ID starting with a6c2586e9cbd95317829fec81618ea376539fb86b8b07e1a5cea67aeaef21ed6 not found: ID does not exist" Oct 30 21:50:10.541335 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.541298 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w"] Oct 30 21:50:10.545326 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.545286 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ebf644d4-59cf567954-tps4w"] Oct 30 21:50:10.630174 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.630141 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45"] Oct 30 21:50:10.630436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.630393 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" podUID="aaade99c-d02b-401c-b71f-77bb257cb7da" containerName="maas-api" containerID="cri-o://d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6" gracePeriod=30 Oct 30 21:50:10.902459 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:10.899101 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:50:11.033150 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.033116 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-metrics-certs\") pod \"aaade99c-d02b-401c-b71f-77bb257cb7da\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " Oct 30 21:50:11.033351 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.033170 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls\") pod \"aaade99c-d02b-401c-b71f-77bb257cb7da\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " Oct 30 21:50:11.033351 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.033306 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaade99c-d02b-401c-b71f-77bb257cb7da-kube-api-access-42jcw\" (UniqueName: \"kubernetes.io/projected/aaade99c-d02b-401c-b71f-77bb257cb7da-kube-api-access-42jcw\") pod \"aaade99c-d02b-401c-b71f-77bb257cb7da\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " Oct 30 21:50:11.033351 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.033332 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaade99c-d02b-401c-b71f-77bb257cb7da-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaade99c-d02b-401c-b71f-77bb257cb7da-openshift-service-ca\") pod \"aaade99c-d02b-401c-b71f-77bb257cb7da\" (UID: \"aaade99c-d02b-401c-b71f-77bb257cb7da\") " Oct 30 21:50:11.033826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.033792 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaade99c-d02b-401c-b71f-77bb257cb7da-openshift-service-ca" pod "aaade99c-d02b-401c-b71f-77bb257cb7da" (UID: "aaade99c-d02b-401c-b71f-77bb257cb7da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:11.035619 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.035588 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaade99c-d02b-401c-b71f-77bb257cb7da-kube-api-access-42jcw" pod "aaade99c-d02b-401c-b71f-77bb257cb7da" (UID: "aaade99c-d02b-401c-b71f-77bb257cb7da"). InnerVolumeSpecName "kube-api-access-42jcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:11.035865 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.035848 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls" pod "aaade99c-d02b-401c-b71f-77bb257cb7da" (UID: "aaade99c-d02b-401c-b71f-77bb257cb7da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:11.035948 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.035921 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-metrics-certs" pod "aaade99c-d02b-401c-b71f-77bb257cb7da" (UID: "aaade99c-d02b-401c-b71f-77bb257cb7da"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:11.134920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.134873 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42jcw\" (UniqueName: \"kubernetes.io/projected/aaade99c-d02b-401c-b71f-77bb257cb7da-kube-api-access-42jcw\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.134920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.134910 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaade99c-d02b-401c-b71f-77bb257cb7da-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.134920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.134921 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.134920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.134930 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaade99c-d02b-401c-b71f-77bb257cb7da-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:11.394873 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.394839 5139 generic.go:356] "Generic (PLEG): container finished" podID="aaade99c-d02b-401c-b71f-77bb257cb7da" containerID="d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6" exitCode=0 Oct 30 21:50:11.395062 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.394886 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" Oct 30 21:50:11.395062 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.394890 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" event={"ID":"aaade99c-d02b-401c-b71f-77bb257cb7da","Type":"ContainerDied","Data":"d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6"} Oct 30 21:50:11.395062 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.395012 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45" event={"ID":"aaade99c-d02b-401c-b71f-77bb257cb7da","Type":"ContainerDied","Data":"81aaa4de201ed75a9e7d626bbb53b5df32d39942116c40bd7e772d6bd8ff0a5a"} Oct 30 21:50:11.395062 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.395036 5139 scope.go:122] "RemoveContainer" containerID="d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6" Oct 30 21:50:11.405069 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.405047 5139 scope.go:122] "RemoveContainer" containerID="d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6" Oct 30 21:50:11.405373 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:11.405354 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6\": container with ID starting with d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6 not found: ID does not exist" containerID="d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6" Oct 30 21:50:11.405459 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.405380 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6"} err="failed to get container status \"d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6\": rpc error: code = NotFound desc = could not find container \"d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6\": container with ID starting with d38cc1df047dda749e38f0be9359e65c76b943a928ad09f9572bdd882b6647f6 not found: ID does not exist" Oct 30 21:50:11.541449 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.541397 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="386a0ef0-71a7-4aa0-a18d-d46318876bc7" path="/var/lib/kubelet/pods/386a0ef0-71a7-4aa0-a18d-d46318876bc7/volumes" Oct 30 21:50:11.547959 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.547927 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45"] Oct 30 21:50:11.555138 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:11.555109 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-09d8fb9b-7c6f8c6785-z4h45"] Oct 30 21:50:13.152831 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:13.152793 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf"] Oct 30 21:50:13.153318 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:13.153049 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" containerID="cri-o://15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2" gracePeriod=30 Oct 30 21:50:13.386344 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:13.386307 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk"] Oct 30 21:50:13.386690 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:13.386635 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" containerID="cri-o://24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30" gracePeriod=30 Oct 30 21:50:13.544147 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:13.544101 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaade99c-d02b-401c-b71f-77bb257cb7da" path="/var/lib/kubelet/pods/aaade99c-d02b-401c-b71f-77bb257cb7da/volumes" Oct 30 21:50:14.206315 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:14.206273 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv"] Oct 30 21:50:14.324700 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:14.324663 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc"] Oct 30 21:50:18.404270 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.404247 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:50:18.434895 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.434807 5139 generic.go:356] "Generic (PLEG): container finished" podID="6511034d-4cf2-4082-9613-e8896dfece70" containerID="15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2" exitCode=0 Oct 30 21:50:18.434895 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.434885 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" Oct 30 21:50:18.435067 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.434888 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" event={"ID":"6511034d-4cf2-4082-9613-e8896dfece70","Type":"ContainerDied","Data":"15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2"} Oct 30 21:50:18.435067 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.435040 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf" event={"ID":"6511034d-4cf2-4082-9613-e8896dfece70","Type":"ContainerDied","Data":"8f038c82ce09b6e83d36dd74d0f9f8ea5b28d34481224227f9bedc1e37b18d69"} Oct 30 21:50:18.435067 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.435065 5139 scope.go:122] "RemoveContainer" containerID="15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2" Oct 30 21:50:18.500406 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.500384 5139 scope.go:122] "RemoveContainer" containerID="8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998" Oct 30 21:50:18.510378 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510351 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-kube-api-access-7gv5x\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-kube-api-access-7gv5x\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510560 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510391 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-istio-token\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-istio-token\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510560 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510430 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-certs\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510560 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510459 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-data\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510560 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510492 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-envoy\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510560 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510542 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-socket\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510845 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510644 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6511034d-4cf2-4082-9613-e8896dfece70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511034d-4cf2-4082-9613-e8896dfece70-istio-podinfo\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510845 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510698 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-credential-socket\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510845 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510784 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6511034d-4cf2-4082-9613-e8896dfece70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511034d-4cf2-4082-9613-e8896dfece70-istiod-ca-cert\") pod \"6511034d-4cf2-4082-9613-e8896dfece70\" (UID: \"6511034d-4cf2-4082-9613-e8896dfece70\") " Oct 30 21:50:18.510989 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510943 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-certs" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.511053 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.510965 5139 scope.go:122] "RemoveContainer" containerID="15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2" Oct 30 21:50:18.511169 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.511135 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-socket" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.511268 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.511185 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.511480 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:18.511446 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2\": container with ID starting with 15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2 not found: ID does not exist" containerID="15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2" Oct 30 21:50:18.511626 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.511492 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2"} err="failed to get container status \"15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2\": rpc error: code = NotFound desc = could not find container \"15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2\": container with ID starting with 15de31928afdef1472806db77be25446598ecd81b1233eaff698ae54344d52a2 not found: ID does not exist" Oct 30 21:50:18.511626 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.511518 5139 scope.go:122] "RemoveContainer" containerID="8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998" Oct 30 21:50:18.511811 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.511615 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-credential-socket" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.511929 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:18.511821 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998\": container with ID starting with 8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998 not found: ID does not exist" containerID="8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998" Oct 30 21:50:18.511929 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.511855 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998"} err="failed to get container status \"8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998\": rpc error: code = NotFound desc = could not find container \"8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998\": container with ID starting with 8293ebffd0c60e6764ef64b1ab12491c1094c84a4a7c5429da78dba79f55b998 not found: ID does not exist" Oct 30 21:50:18.512111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.512088 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6511034d-4cf2-4082-9613-e8896dfece70-istiod-ca-cert" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:18.512196 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.512124 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-data" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.512967 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.512935 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-kube-api-access-7gv5x" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "kube-api-access-7gv5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:18.514458 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.514434 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-istio-token" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:18.514721 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.514573 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-envoy" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.514825 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.514805 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6511034d-4cf2-4082-9613-e8896dfece70-istio-podinfo" pod "6511034d-4cf2-4082-9613-e8896dfece70" (UID: "6511034d-4cf2-4082-9613-e8896dfece70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:18.612772 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.612732 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gv5x\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-kube-api-access-7gv5x\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.612772 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.612768 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6511034d-4cf2-4082-9613-e8896dfece70-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.612772 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.612779 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.612992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.612788 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.612992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.612797 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.612992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.612806 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6511034d-4cf2-4082-9613-e8896dfece70-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.612992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.612814 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6511034d-4cf2-4082-9613-e8896dfece70-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.612992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.612821 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6511034d-4cf2-4082-9613-e8896dfece70-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.625785 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.625760 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:50:18.713914 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.713827 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-credential-socket\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.713914 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.713876 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-podinfo\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.713914 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.713904 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-data\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.714160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.713946 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-certs\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.714160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.713987 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-socket\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.714160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714016 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-token\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-token\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.714160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714073 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istiod-ca-cert\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.714160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714104 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-envoy\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.714160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714135 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-credential-socket" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.714501 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714187 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-kube-api-access-8r7bf\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-kube-api-access-8r7bf\") pod \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\" (UID: \"0da58100-bf31-4805-94dd-c5a34dfd3ff9\") " Oct 30 21:50:18.714501 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714286 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-certs" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.714501 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714330 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-socket" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.714501 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714295 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-data" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.714695 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714514 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istiod-ca-cert" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:18.715017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714777 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.715017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714805 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.715017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714822 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.715017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714838 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.715017 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.714853 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.716366 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.716344 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-kube-api-access-8r7bf" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "kube-api-access-8r7bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:18.716690 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.716663 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-envoy" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:50:18.716769 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.716667 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-podinfo" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 21:50:18.716769 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.716694 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-token" pod "0da58100-bf31-4805-94dd-c5a34dfd3ff9" (UID: "0da58100-bf31-4805-94dd-c5a34dfd3ff9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:18.816353 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.816315 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.816353 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.816345 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.816353 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.816355 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8r7bf\" (UniqueName: \"kubernetes.io/projected/0da58100-bf31-4805-94dd-c5a34dfd3ff9-kube-api-access-8r7bf\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.816617 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.816365 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0da58100-bf31-4805-94dd-c5a34dfd3ff9-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:18.850777 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.850740 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf"] Oct 30 21:50:18.857446 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:18.857401 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ebf644d4-openshift-default-f64795575-8vdnf"] Oct 30 21:50:19.207156 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.207105 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" podUID="67b9354c-5b48-4154-b77d-c30adb5ad520" containerName="payload-processing" containerID="cri-o://98b3a7b3f05f9e146b75a7b03e91fcd1a85bfc40339292999b2954c5e1c8a435" gracePeriod=25 Oct 30 21:50:19.325492 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.325454 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" podUID="b1392c96-2823-4563-97ab-cadb52cf94f9" containerName="payload-pre-processing" containerID="cri-o://5e251ffe4ac396be6d821e521a2b85df6d9fbb26aff38a94ed846cafedfdd851" gracePeriod=25 Oct 30 21:50:19.443174 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.443139 5139 generic.go:356] "Generic (PLEG): container finished" podID="b1392c96-2823-4563-97ab-cadb52cf94f9" containerID="5e251ffe4ac396be6d821e521a2b85df6d9fbb26aff38a94ed846cafedfdd851" exitCode=0 Oct 30 21:50:19.443627 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.443207 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" event={"ID":"b1392c96-2823-4563-97ab-cadb52cf94f9","Type":"ContainerDied","Data":"5e251ffe4ac396be6d821e521a2b85df6d9fbb26aff38a94ed846cafedfdd851"} Oct 30 21:50:19.444674 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.444651 5139 generic.go:356] "Generic (PLEG): container finished" podID="67b9354c-5b48-4154-b77d-c30adb5ad520" containerID="98b3a7b3f05f9e146b75a7b03e91fcd1a85bfc40339292999b2954c5e1c8a435" exitCode=0 Oct 30 21:50:19.444788 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.444721 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" event={"ID":"67b9354c-5b48-4154-b77d-c30adb5ad520","Type":"ContainerDied","Data":"98b3a7b3f05f9e146b75a7b03e91fcd1a85bfc40339292999b2954c5e1c8a435"} Oct 30 21:50:19.444788 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.444781 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" event={"ID":"67b9354c-5b48-4154-b77d-c30adb5ad520","Type":"ContainerDied","Data":"ccfb6ebb48ec5c91805b396ede4f04e098b980d96a6d108e0030dbce7000d8c4"} Oct 30 21:50:19.444894 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.444806 5139 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccfb6ebb48ec5c91805b396ede4f04e098b980d96a6d108e0030dbce7000d8c4" Oct 30 21:50:19.447644 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.447619 5139 generic.go:356] "Generic (PLEG): container finished" podID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerID="24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30" exitCode=0 Oct 30 21:50:19.447768 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.447680 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" Oct 30 21:50:19.447768 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.447705 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" event={"ID":"0da58100-bf31-4805-94dd-c5a34dfd3ff9","Type":"ContainerDied","Data":"24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30"} Oct 30 21:50:19.447886 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.447774 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk" event={"ID":"0da58100-bf31-4805-94dd-c5a34dfd3ff9","Type":"ContainerDied","Data":"f09e2899c68b6dc36bd44a86118db6bdba45452afaa76c755dd61bf9f6fd5751"} Oct 30 21:50:19.447886 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.447807 5139 scope.go:122] "RemoveContainer" containerID="24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30" Oct 30 21:50:19.454536 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.454462 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:50:19.459068 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.459049 5139 scope.go:122] "RemoveContainer" containerID="adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467" Oct 30 21:50:19.467209 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.467185 5139 scope.go:122] "RemoveContainer" containerID="24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30" Oct 30 21:50:19.467546 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:19.467525 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30\": container with ID starting with 24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30 not found: ID does not exist" containerID="24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30" Oct 30 21:50:19.467598 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.467557 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30"} err="failed to get container status \"24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30\": rpc error: code = NotFound desc = could not find container \"24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30\": container with ID starting with 24ee3a416c79b1a6c2f3e3a1326e1b82e02336ae82346f293248aa763b571b30 not found: ID does not exist" Oct 30 21:50:19.467598 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.467576 5139 scope.go:122] "RemoveContainer" containerID="adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467" Oct 30 21:50:19.467861 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:19.467840 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467\": container with ID starting with adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467 not found: ID does not exist" containerID="adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467" Oct 30 21:50:19.467915 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.467873 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467"} err="failed to get container status \"adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467\": rpc error: code = NotFound desc = could not find container \"adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467\": container with ID starting with adb3ea28623fc78dabff69e6f895fbb975b0c4f777b95862f873c4f90b36f467 not found: ID does not exist" Oct 30 21:50:19.543392 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.543359 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6511034d-4cf2-4082-9613-e8896dfece70" path="/var/lib/kubelet/pods/6511034d-4cf2-4082-9613-e8896dfece70/volumes" Oct 30 21:50:19.566023 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.565985 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:50:19.623703 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.623673 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b9354c-5b48-4154-b77d-c30adb5ad520-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b9354c-5b48-4154-b77d-c30adb5ad520-plugins-config-volume\") pod \"67b9354c-5b48-4154-b77d-c30adb5ad520\" (UID: \"67b9354c-5b48-4154-b77d-c30adb5ad520\") " Oct 30 21:50:19.623881 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.623755 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b9354c-5b48-4154-b77d-c30adb5ad520-kube-api-access-sm4p9\" (UniqueName: \"kubernetes.io/projected/67b9354c-5b48-4154-b77d-c30adb5ad520-kube-api-access-sm4p9\") pod \"67b9354c-5b48-4154-b77d-c30adb5ad520\" (UID: \"67b9354c-5b48-4154-b77d-c30adb5ad520\") " Oct 30 21:50:19.624208 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.624175 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b9354c-5b48-4154-b77d-c30adb5ad520-plugins-config-volume" pod "67b9354c-5b48-4154-b77d-c30adb5ad520" (UID: "67b9354c-5b48-4154-b77d-c30adb5ad520"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:19.626003 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.625972 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b9354c-5b48-4154-b77d-c30adb5ad520-kube-api-access-sm4p9" pod "67b9354c-5b48-4154-b77d-c30adb5ad520" (UID: "67b9354c-5b48-4154-b77d-c30adb5ad520"). InnerVolumeSpecName "kube-api-access-sm4p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:19.714233 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.714148 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk"] Oct 30 21:50:19.717687 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.717657 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-09d8fb9b-openshift-default-678f87597-59fmk"] Oct 30 21:50:19.724309 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.724285 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1392c96-2823-4563-97ab-cadb52cf94f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1392c96-2823-4563-97ab-cadb52cf94f9-plugins-config-volume\") pod \"b1392c96-2823-4563-97ab-cadb52cf94f9\" (UID: \"b1392c96-2823-4563-97ab-cadb52cf94f9\") " Oct 30 21:50:19.724431 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.724319 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1392c96-2823-4563-97ab-cadb52cf94f9-kube-api-access-zzhgn\" (UniqueName: \"kubernetes.io/projected/b1392c96-2823-4563-97ab-cadb52cf94f9-kube-api-access-zzhgn\") pod \"b1392c96-2823-4563-97ab-cadb52cf94f9\" (UID: \"b1392c96-2823-4563-97ab-cadb52cf94f9\") " Oct 30 21:50:19.724560 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.724549 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67b9354c-5b48-4154-b77d-c30adb5ad520-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.724604 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.724562 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm4p9\" (UniqueName: \"kubernetes.io/projected/67b9354c-5b48-4154-b77d-c30adb5ad520-kube-api-access-sm4p9\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.724712 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.724686 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1392c96-2823-4563-97ab-cadb52cf94f9-plugins-config-volume" pod "b1392c96-2823-4563-97ab-cadb52cf94f9" (UID: "b1392c96-2823-4563-97ab-cadb52cf94f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:19.726557 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.726533 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1392c96-2823-4563-97ab-cadb52cf94f9-kube-api-access-zzhgn" pod "b1392c96-2823-4563-97ab-cadb52cf94f9" (UID: "b1392c96-2823-4563-97ab-cadb52cf94f9"). InnerVolumeSpecName "kube-api-access-zzhgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:19.805600 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:19.805552 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:50:19.825684 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.825651 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1392c96-2823-4563-97ab-cadb52cf94f9-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:19.825684 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:19.825681 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzhgn\" (UniqueName: \"kubernetes.io/projected/b1392c96-2823-4563-97ab-cadb52cf94f9-kube-api-access-zzhgn\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:20.454398 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:20.454309 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" Oct 30 21:50:20.454398 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:20.454334 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc" event={"ID":"b1392c96-2823-4563-97ab-cadb52cf94f9","Type":"ContainerDied","Data":"edb6e0ea2fb8ff79bd1d600eaf7ed53bcfddcd2d1c076b7ce5ac92e0edaeee25"} Oct 30 21:50:20.454398 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:20.454390 5139 scope.go:122] "RemoveContainer" containerID="5e251ffe4ac396be6d821e521a2b85df6d9fbb26aff38a94ed846cafedfdd851" Oct 30 21:50:20.455730 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:20.455614 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv" Oct 30 21:50:20.608029 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:20.607992 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv"] Oct 30 21:50:20.654322 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:20.654286 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a1d107b0-7848ff8d87-gdkvv"] Oct 30 21:50:20.761697 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:20.761668 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc"] Oct 30 21:50:20.765467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:20.765443 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a1d107b0-56ffb8d479-5zgmc"] Oct 30 21:50:21.542063 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:21.542027 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" path="/var/lib/kubelet/pods/0da58100-bf31-4805-94dd-c5a34dfd3ff9/volumes" Oct 30 21:50:21.542545 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:21.542527 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b9354c-5b48-4154-b77d-c30adb5ad520" path="/var/lib/kubelet/pods/67b9354c-5b48-4154-b77d-c30adb5ad520/volumes" Oct 30 21:50:21.542988 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:21.542975 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1392c96-2823-4563-97ab-cadb52cf94f9" path="/var/lib/kubelet/pods/b1392c96-2823-4563-97ab-cadb52cf94f9/volumes" Oct 30 21:50:24.531786 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.531747 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w"] Oct 30 21:50:24.532352 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532330 5139 cpu_manager.go:414] "containerMap: removing container" podUID="b1392c96-2823-4563-97ab-cadb52cf94f9" containerName="payload-pre-processing" Oct 30 21:50:24.532477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532354 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1392c96-2823-4563-97ab-cadb52cf94f9" containerName="payload-pre-processing" Oct 30 21:50:24.532477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532371 5139 cpu_manager.go:414] "containerMap: removing container" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" Oct 30 21:50:24.532477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532379 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" Oct 30 21:50:24.532477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532404 5139 cpu_manager.go:414] "containerMap: removing container" podUID="aaade99c-d02b-401c-b71f-77bb257cb7da" containerName="maas-api" Oct 30 21:50:24.532477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532427 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaade99c-d02b-401c-b71f-77bb257cb7da" containerName="maas-api" Oct 30 21:50:24.532477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532447 5139 cpu_manager.go:414] "containerMap: removing container" podUID="67b9354c-5b48-4154-b77d-c30adb5ad520" containerName="payload-processing" Oct 30 21:50:24.532477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532456 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b9354c-5b48-4154-b77d-c30adb5ad520" containerName="payload-processing" Oct 30 21:50:24.532477 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532473 5139 cpu_manager.go:414] "containerMap: removing container" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532481 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532497 5139 cpu_manager.go:414] "containerMap: removing container" podUID="386a0ef0-71a7-4aa0-a18d-d46318876bc7" containerName="maas-api" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532505 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="386a0ef0-71a7-4aa0-a18d-d46318876bc7" containerName="maas-api" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532518 5139 cpu_manager.go:414] "containerMap: removing container" podUID="eed05a9a-2235-4053-ae44-6049f835490e" containerName="payload-processing" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532526 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed05a9a-2235-4053-ae44-6049f835490e" containerName="payload-processing" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532537 5139 cpu_manager.go:414] "containerMap: removing container" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532544 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532641 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532654 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaade99c-d02b-401c-b71f-77bb257cb7da" containerName="maas-api" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532667 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b9354c-5b48-4154-b77d-c30adb5ad520" containerName="payload-processing" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532677 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532686 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="386a0ef0-71a7-4aa0-a18d-d46318876bc7" containerName="maas-api" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532693 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed05a9a-2235-4053-ae44-6049f835490e" containerName="payload-processing" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532701 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="0da58100-bf31-4805-94dd-c5a34dfd3ff9" containerName="istio-proxy" Oct 30 21:50:24.532932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.532711 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1392c96-2823-4563-97ab-cadb52cf94f9" containerName="payload-pre-processing" Oct 30 21:50:24.535559 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.535530 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.589216 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.589184 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-26743a-openshift-default-dockercfg-lddtx\"" Oct 30 21:50:24.637083 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.637049 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w"] Oct 30 21:50:24.672864 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.672835 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.673060 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.672870 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.673060 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.672907 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.673060 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.673013 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.673060 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.673060 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.673277 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.673171 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.673277 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.673213 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-956ct\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-kube-api-access-956ct\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.673360 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.673286 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.673360 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.673342 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774294 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774328 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774335 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774593 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774360 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774593 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774480 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774593 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774536 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774593 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774578 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774598 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-956ct\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-kube-api-access-956ct\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774629 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774675 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774797 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.774826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.774801 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.775080 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.775054 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.775140 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.775078 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.775443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.775400 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.776890 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.776851 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.778289 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.777397 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.834797 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.834725 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.835855 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.835830 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-956ct\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-kube-api-access-956ct\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.848545 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.848517 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:24.871049 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.871020 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns"] Oct 30 21:50:24.871512 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.871496 5139 cpu_manager.go:414] "containerMap: removing container" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" Oct 30 21:50:24.871611 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.871514 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" Oct 30 21:50:24.871673 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.871660 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="6511034d-4cf2-4082-9613-e8896dfece70" containerName="istio-proxy" Oct 30 21:50:24.874592 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.874560 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.924277 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.924185 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9a5623-openshift-default-dockercfg-gltcf\"" Oct 30 21:50:24.933793 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.932670 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns"] Oct 30 21:50:24.976489 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976455 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.976489 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976492 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.976694 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976531 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.976694 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976627 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.976768 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976713 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.976768 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976730 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.976835 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976785 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.976835 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976811 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd723b38-6cea-4308-bcf3-26369bbfeecc-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:24.976902 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:24.976882 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ppx9\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-kube-api-access-5ppx9\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077540 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077503 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077540 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077540 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd723b38-6cea-4308-bcf3-26369bbfeecc-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077755 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077660 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ppx9\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-kube-api-access-5ppx9\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077755 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077740 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077832 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077764 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077832 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077822 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077930 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077848 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077930 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077902 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.077930 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077905 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.078102 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.077929 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.078188 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.078156 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.078258 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.078183 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.078311 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.078294 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd723b38-6cea-4308-bcf3-26369bbfeecc-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.078364 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.078297 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.080157 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.080132 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.080324 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.080307 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.135507 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.135399 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.135632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.135567 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ppx9\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-kube-api-access-5ppx9\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.186602 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.186569 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:25.275313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.275271 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w"] Oct 30 21:50:25.282123 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:50:25.282093 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc912fda5_cfef_4f0e_b9b4_8f6f6f44aeef.slice/crio-c28eaabe58434297d7bd1be3387b86b4c65d042f2243dc901e3c6a52bdc483fa WatchSource:0}: Error finding container c28eaabe58434297d7bd1be3387b86b4c65d042f2243dc901e3c6a52bdc483fa: Status 404 returned error can't find the container with id c28eaabe58434297d7bd1be3387b86b4c65d042f2243dc901e3c6a52bdc483fa Oct 30 21:50:25.284560 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.284517 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:25.284656 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.284620 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:25.284700 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.284668 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:25.483199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.483104 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" event={"ID":"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef","Type":"ContainerStarted","Data":"7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88"} Oct 30 21:50:25.483199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.483164 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" event={"ID":"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef","Type":"ContainerStarted","Data":"c28eaabe58434297d7bd1be3387b86b4c65d042f2243dc901e3c6a52bdc483fa"} Oct 30 21:50:25.561668 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.561637 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns"] Oct 30 21:50:25.568688 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:50:25.568651 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd723b38_6cea_4308_bcf3_26369bbfeecc.slice/crio-d0531eaf33bbbbb34a562411abf0f9cb059471459219bc5417db8d065e3b9da1 WatchSource:0}: Error finding container d0531eaf33bbbbb34a562411abf0f9cb059471459219bc5417db8d065e3b9da1: Status 404 returned error can't find the container with id d0531eaf33bbbbb34a562411abf0f9cb059471459219bc5417db8d065e3b9da1 Oct 30 21:50:25.571431 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.571384 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:25.571528 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.571469 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:25.571528 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.571503 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:25.598544 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.598497 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" podStartSLOduration=1.59847982 podStartE2EDuration="1.59847982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:25.596133986 +0000 UTC m=+734.187419474" watchObservedRunningTime="2026-10-30 21:50:25.59847982 +0000 UTC m=+734.189765307" Oct 30 21:50:25.849489 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:25.849457 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:26.489466 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:26.489429 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" event={"ID":"dd723b38-6cea-4308-bcf3-26369bbfeecc","Type":"ContainerStarted","Data":"d0091646eb464ed633ffd8710361fc7d45eced785f55eda5876bd9b4c3e896b8"} Oct 30 21:50:26.489666 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:26.489509 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" event={"ID":"dd723b38-6cea-4308-bcf3-26369bbfeecc","Type":"ContainerStarted","Data":"d0531eaf33bbbbb34a562411abf0f9cb059471459219bc5417db8d065e3b9da1"} Oct 30 21:50:26.652869 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:26.652808 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" podStartSLOduration=2.6527947039999997 podStartE2EDuration="2.652794704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:26.65091866 +0000 UTC m=+735.242204147" watchObservedRunningTime="2026-10-30 21:50:26.652794704 +0000 UTC m=+735.244080246" Oct 30 21:50:26.855273 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:26.855194 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:27.187218 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:27.187133 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:27.192634 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:27.192609 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:27.494919 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:27.494882 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:27.495092 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:27.494949 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:27.495960 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:27.495927 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:50:27.496075 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:27.496034 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:50:31.318885 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.318849 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv"] Oct 30 21:50:31.319299 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.319039 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" podUID="ce0a5e91-3fde-415a-a14d-5f128365afd4" containerName="maas-api" containerID="cri-o://4711040cbb3dcc37920b4cd71277bcd3e2f01cfece7ddcacb78048f3ad74f96a" gracePeriod=30 Oct 30 21:50:31.518009 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.517970 5139 generic.go:356] "Generic (PLEG): container finished" podID="ce0a5e91-3fde-415a-a14d-5f128365afd4" containerID="4711040cbb3dcc37920b4cd71277bcd3e2f01cfece7ddcacb78048f3ad74f96a" exitCode=0 Oct 30 21:50:31.518168 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.518044 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" event={"ID":"ce0a5e91-3fde-415a-a14d-5f128365afd4","Type":"ContainerDied","Data":"4711040cbb3dcc37920b4cd71277bcd3e2f01cfece7ddcacb78048f3ad74f96a"} Oct 30 21:50:31.572227 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.572171 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:50:31.644584 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.644552 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls\") pod \"ce0a5e91-3fde-415a-a14d-5f128365afd4\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " Oct 30 21:50:31.644584 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.644594 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-metrics-certs\") pod \"ce0a5e91-3fde-415a-a14d-5f128365afd4\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " Oct 30 21:50:31.644820 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.644658 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0a5e91-3fde-415a-a14d-5f128365afd4-kube-api-access-hbsqt\" (UniqueName: \"kubernetes.io/projected/ce0a5e91-3fde-415a-a14d-5f128365afd4-kube-api-access-hbsqt\") pod \"ce0a5e91-3fde-415a-a14d-5f128365afd4\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " Oct 30 21:50:31.644820 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.644803 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0a5e91-3fde-415a-a14d-5f128365afd4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0a5e91-3fde-415a-a14d-5f128365afd4-openshift-service-ca\") pod \"ce0a5e91-3fde-415a-a14d-5f128365afd4\" (UID: \"ce0a5e91-3fde-415a-a14d-5f128365afd4\") " Oct 30 21:50:31.645385 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.645332 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0a5e91-3fde-415a-a14d-5f128365afd4-openshift-service-ca" pod "ce0a5e91-3fde-415a-a14d-5f128365afd4" (UID: "ce0a5e91-3fde-415a-a14d-5f128365afd4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:31.646956 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.646927 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0a5e91-3fde-415a-a14d-5f128365afd4-kube-api-access-hbsqt" pod "ce0a5e91-3fde-415a-a14d-5f128365afd4" (UID: "ce0a5e91-3fde-415a-a14d-5f128365afd4"). InnerVolumeSpecName "kube-api-access-hbsqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:31.647186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.647160 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls" pod "ce0a5e91-3fde-415a-a14d-5f128365afd4" (UID: "ce0a5e91-3fde-415a-a14d-5f128365afd4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:31.647271 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.647190 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-metrics-certs" pod "ce0a5e91-3fde-415a-a14d-5f128365afd4" (UID: "ce0a5e91-3fde-415a-a14d-5f128365afd4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:31.746149 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.746115 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbsqt\" (UniqueName: \"kubernetes.io/projected/ce0a5e91-3fde-415a-a14d-5f128365afd4-kube-api-access-hbsqt\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:31.746149 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.746144 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce0a5e91-3fde-415a-a14d-5f128365afd4-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:31.746149 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.746156 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:31.746360 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:31.746164 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce0a5e91-3fde-415a-a14d-5f128365afd4-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:32.524185 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:32.524153 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" event={"ID":"ce0a5e91-3fde-415a-a14d-5f128365afd4","Type":"ContainerDied","Data":"7be614f312179a5042929437d3a12d323642ab541ea0c9b5c2da98a837b76e03"} Oct 30 21:50:32.524666 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:32.524196 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv" Oct 30 21:50:32.524666 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:32.524249 5139 scope.go:122] "RemoveContainer" containerID="4711040cbb3dcc37920b4cd71277bcd3e2f01cfece7ddcacb78048f3ad74f96a" Oct 30 21:50:32.678845 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:32.678806 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv"] Oct 30 21:50:32.725274 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:32.725243 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a1d107b0-784ddf6475-znhkv"] Oct 30 21:50:33.541740 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:33.541710 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0a5e91-3fde-415a-a14d-5f128365afd4" path="/var/lib/kubelet/pods/ce0a5e91-3fde-415a-a14d-5f128365afd4/volumes" Oct 30 21:50:34.935447 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.935402 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x"] Oct 30 21:50:34.935827 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.935814 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ce0a5e91-3fde-415a-a14d-5f128365afd4" containerName="maas-api" Oct 30 21:50:34.935887 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.935827 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0a5e91-3fde-415a-a14d-5f128365afd4" containerName="maas-api" Oct 30 21:50:34.935919 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.935890 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0a5e91-3fde-415a-a14d-5f128365afd4" containerName="maas-api" Oct 30 21:50:34.939095 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.939076 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981297 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981246 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zqfq\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-kube-api-access-8zqfq\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981452 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981337 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981452 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981371 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981452 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981440 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981572 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981507 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981572 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981546 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981641 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981603 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981679 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981651 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.981714 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.981688 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:34.990839 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:34.990808 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-56f3fe-openshift-default-dockercfg-48wbx\"" Oct 30 21:50:35.038934 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.038893 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x"] Oct 30 21:50:35.083000 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.082970 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083000 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083001 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083033 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083060 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083082 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083104 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zqfq\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-kube-api-access-8zqfq\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083196 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083469 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083240 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083469 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083293 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083469 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083400 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083469 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083457 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083674 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083542 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.083674 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.083599 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.084088 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.084064 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.085594 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.085571 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.085706 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.085685 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.144762 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.144722 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.144918 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.144821 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zqfq\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-kube-api-access-8zqfq\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.250520 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.250489 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:35.670781 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.670691 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x"] Oct 30 21:50:35.676829 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:50:35.676802 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e166eda_5b4a_4d02_9d07_6d577fd9706a.slice/crio-e0084bdb3d9b6b6a505e9c2736ecf0e4a60083346f52231c8e252d8dc393d260 WatchSource:0}: Error finding container e0084bdb3d9b6b6a505e9c2736ecf0e4a60083346f52231c8e252d8dc393d260: Status 404 returned error can't find the container with id e0084bdb3d9b6b6a505e9c2736ecf0e4a60083346f52231c8e252d8dc393d260 Oct 30 21:50:35.678931 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.678899 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:35.678993 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.678970 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:35.679030 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:35.678998 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:36.546321 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:36.546285 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" event={"ID":"9e166eda-5b4a-4d02-9d07-6d577fd9706a","Type":"ContainerStarted","Data":"5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e"} Oct 30 21:50:36.546844 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:36.546337 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" event={"ID":"9e166eda-5b4a-4d02-9d07-6d577fd9706a","Type":"ContainerStarted","Data":"e0084bdb3d9b6b6a505e9c2736ecf0e4a60083346f52231c8e252d8dc393d260"} Oct 30 21:50:36.707324 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:36.707262 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" podStartSLOduration=2.707245827 podStartE2EDuration="2.707245827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:36.705801263 +0000 UTC m=+745.297086752" watchObservedRunningTime="2026-10-30 21:50:36.707245827 +0000 UTC m=+745.298531391" Oct 30 21:50:37.251657 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.251618 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:37.256986 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.256961 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:37.354166 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.354134 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg"] Oct 30 21:50:37.354404 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.354380 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" podUID="9d9ddc82-b923-40c2-8d90-c172f0494fe1" containerName="maas-api" containerID="cri-o://8eaf7c8ecf9eda73ed9b95134001d5d68e98c5416350a0dfa389125e4fbcd8b1" gracePeriod=30 Oct 30 21:50:37.552692 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.552660 5139 generic.go:356] "Generic (PLEG): container finished" podID="9d9ddc82-b923-40c2-8d90-c172f0494fe1" containerID="8eaf7c8ecf9eda73ed9b95134001d5d68e98c5416350a0dfa389125e4fbcd8b1" exitCode=0 Oct 30 21:50:37.553157 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.552734 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" event={"ID":"9d9ddc82-b923-40c2-8d90-c172f0494fe1","Type":"ContainerDied","Data":"8eaf7c8ecf9eda73ed9b95134001d5d68e98c5416350a0dfa389125e4fbcd8b1"} Oct 30 21:50:37.553157 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.553137 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:37.554199 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.554174 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:50:37.597111 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.597085 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:50:37.713323 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.713291 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d9ddc82-b923-40c2-8d90-c172f0494fe1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ddc82-b923-40c2-8d90-c172f0494fe1-openshift-service-ca\") pod \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " Oct 30 21:50:37.713623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.713352 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d9ddc82-b923-40c2-8d90-c172f0494fe1-kube-api-access-5r5rs\" (UniqueName: \"kubernetes.io/projected/9d9ddc82-b923-40c2-8d90-c172f0494fe1-kube-api-access-5r5rs\") pod \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " Oct 30 21:50:37.713623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.713476 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-metrics-certs\") pod \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " Oct 30 21:50:37.713623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.713531 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls\") pod \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\" (UID: \"9d9ddc82-b923-40c2-8d90-c172f0494fe1\") " Oct 30 21:50:37.713826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.713734 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d9ddc82-b923-40c2-8d90-c172f0494fe1-openshift-service-ca" pod "9d9ddc82-b923-40c2-8d90-c172f0494fe1" (UID: "9d9ddc82-b923-40c2-8d90-c172f0494fe1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 21:50:37.713882 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.713854 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d9ddc82-b923-40c2-8d90-c172f0494fe1-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:37.715611 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.715578 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d9ddc82-b923-40c2-8d90-c172f0494fe1-kube-api-access-5r5rs" pod "9d9ddc82-b923-40c2-8d90-c172f0494fe1" (UID: "9d9ddc82-b923-40c2-8d90-c172f0494fe1"). InnerVolumeSpecName "kube-api-access-5r5rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:50:37.715611 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.715604 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-metrics-certs" pod "9d9ddc82-b923-40c2-8d90-c172f0494fe1" (UID: "9d9ddc82-b923-40c2-8d90-c172f0494fe1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:37.715805 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.715650 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls" pod "9d9ddc82-b923-40c2-8d90-c172f0494fe1" (UID: "9d9ddc82-b923-40c2-8d90-c172f0494fe1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:50:37.814614 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.814543 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r5rs\" (UniqueName: \"kubernetes.io/projected/9d9ddc82-b923-40c2-8d90-c172f0494fe1-kube-api-access-5r5rs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:37.814614 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.814570 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:37.814614 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:37.814581 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d9ddc82-b923-40c2-8d90-c172f0494fe1-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:50:38.558085 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:38.558054 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" Oct 30 21:50:38.558626 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:38.558086 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg" event={"ID":"9d9ddc82-b923-40c2-8d90-c172f0494fe1","Type":"ContainerDied","Data":"8794870cbf4453efe3b040e5547fa08f7ddca332fee7fd5daedbb11bdc7d2321"} Oct 30 21:50:38.558626 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:38.558158 5139 scope.go:122] "RemoveContainer" containerID="8eaf7c8ecf9eda73ed9b95134001d5d68e98c5416350a0dfa389125e4fbcd8b1" Oct 30 21:50:38.726867 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:38.726832 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg"] Oct 30 21:50:38.772817 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:38.772784 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-16c49820-74bc47897c-57mbg"] Oct 30 21:50:39.545470 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:39.545432 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d9ddc82-b923-40c2-8d90-c172f0494fe1" path="/var/lib/kubelet/pods/9d9ddc82-b923-40c2-8d90-c172f0494fe1/volumes" Oct 30 21:50:40.177114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.177077 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm"] Oct 30 21:50:40.177500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.177478 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9d9ddc82-b923-40c2-8d90-c172f0494fe1" containerName="maas-api" Oct 30 21:50:40.177500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.177490 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d9ddc82-b923-40c2-8d90-c172f0494fe1" containerName="maas-api" Oct 30 21:50:40.177600 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.177549 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d9ddc82-b923-40c2-8d90-c172f0494fe1" containerName="maas-api" Oct 30 21:50:40.180651 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.180628 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.229837 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.229801 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-9a5623-serving-cert\"" Oct 30 21:50:40.240932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.240895 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-metrics-certs\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.241073 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.241057 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.241117 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.241101 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.241271 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.241246 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktjvz\" (UniqueName: \"kubernetes.io/projected/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-kube-api-access-ktjvz\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.277513 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.277476 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm"] Oct 30 21:50:40.341788 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.341754 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.342028 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.341818 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.342028 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.341855 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktjvz\" (UniqueName: \"kubernetes.io/projected/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-kube-api-access-ktjvz\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.342028 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.341913 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-metrics-certs\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.342028 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:40.341984 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-serving-cert: secret "maas-api-e2e-worker-w0-9a5623-serving-cert" not found Oct 30 21:50:40.342248 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:40.342057 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls podName:569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e nodeName:}" failed. No retries permitted until 2026-10-30 21:50:40.842034627 +0000 UTC m=+749.433320096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls") pod "maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" (UID: "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e") : secret "maas-api-e2e-worker-w0-9a5623-serving-cert" not found Oct 30 21:50:40.342471 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.342451 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.344260 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.344233 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-metrics-certs\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.415212 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.415184 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktjvz\" (UniqueName: \"kubernetes.io/projected/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-kube-api-access-ktjvz\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.847188 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.847152 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:40.849573 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:40.849550 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls\") pod \"maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:41.092719 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:41.092686 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:41.464689 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:41.464610 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm"] Oct 30 21:50:41.470917 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:50:41.470891 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569d93b0_ea0b_4a4a_8d49_8e7e6af08a6e.slice/crio-5507b1b654caf4cdd55a60bcdfe63007fdccbf7f73c79f54f560eef5070bd119 WatchSource:0}: Error finding container 5507b1b654caf4cdd55a60bcdfe63007fdccbf7f73c79f54f560eef5070bd119: Status 404 returned error can't find the container with id 5507b1b654caf4cdd55a60bcdfe63007fdccbf7f73c79f54f560eef5070bd119 Oct 30 21:50:41.472710 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:41.472691 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 21:50:41.575447 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:41.575403 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" event={"ID":"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e","Type":"ContainerStarted","Data":"5507b1b654caf4cdd55a60bcdfe63007fdccbf7f73c79f54f560eef5070bd119"} Oct 30 21:50:42.581084 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:42.581051 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" event={"ID":"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e","Type":"ContainerStarted","Data":"995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42"} Oct 30 21:50:42.581500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:42.581118 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:42.695313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:42.695216 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" podStartSLOduration=1.760452523 podStartE2EDuration="2.695202983s" totalImagesPullingTime="934.75046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:42.693564706 +0000 UTC m=+751.284850193" watchObservedRunningTime="2026-10-30 21:50:42.695202983 +0000 UTC m=+751.286488469" Oct 30 21:50:48.597868 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:48.597837 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 21:50:49.804197 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:49.804156 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:50:55.055728 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.055691 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss"] Oct 30 21:50:55.058403 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.058384 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.107620 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.107582 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-56f3fe-serving-cert\"" Oct 30 21:50:55.112108 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.112075 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2dtl\" (UniqueName: \"kubernetes.io/projected/f60eb6af-5bbd-4a80-b835-60edc3466d68-kube-api-access-r2dtl\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.112246 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.112126 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.112246 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.112224 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eb6af-5bbd-4a80-b835-60edc3466d68-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.112354 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.112266 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-metrics-certs\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.155549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.155509 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss"] Oct 30 21:50:55.213657 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.213616 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2dtl\" (UniqueName: \"kubernetes.io/projected/f60eb6af-5bbd-4a80-b835-60edc3466d68-kube-api-access-r2dtl\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.213862 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.213672 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.213862 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.213711 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eb6af-5bbd-4a80-b835-60edc3466d68-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.213862 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.213743 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-metrics-certs\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.213862 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:55.213846 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-serving-cert: secret "maas-api-e2e-worker-w2-56f3fe-serving-cert" not found Oct 30 21:50:55.214090 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:50:55.213940 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls podName:f60eb6af-5bbd-4a80-b835-60edc3466d68 nodeName:}" failed. No retries permitted until 2026-10-30 21:50:55.713916339 +0000 UTC m=+764.305201818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls") pod "maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" (UID: "f60eb6af-5bbd-4a80-b835-60edc3466d68") : secret "maas-api-e2e-worker-w2-56f3fe-serving-cert" not found Oct 30 21:50:55.214440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.214390 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eb6af-5bbd-4a80-b835-60edc3466d68-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.216279 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.216263 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-metrics-certs\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.290285 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.290252 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2dtl\" (UniqueName: \"kubernetes.io/projected/f60eb6af-5bbd-4a80-b835-60edc3466d68-kube-api-access-r2dtl\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.719062 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.719015 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.721544 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.721520 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls\") pod \"maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:55.995016 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:55.994983 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:56.367494 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:56.362739 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss"] Oct 30 21:50:56.391011 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:50:56.389985 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60eb6af_5bbd_4a80_b835_60edc3466d68.slice/crio-334fd158ad1876c0be41687ea5f6e3f3ff517d977e5e596192ffd3d9f1849d1c WatchSource:0}: Error finding container 334fd158ad1876c0be41687ea5f6e3f3ff517d977e5e596192ffd3d9f1849d1c: Status 404 returned error can't find the container with id 334fd158ad1876c0be41687ea5f6e3f3ff517d977e5e596192ffd3d9f1849d1c Oct 30 21:50:56.656323 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:56.656235 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" event={"ID":"f60eb6af-5bbd-4a80-b835-60edc3466d68","Type":"ContainerStarted","Data":"334fd158ad1876c0be41687ea5f6e3f3ff517d977e5e596192ffd3d9f1849d1c"} Oct 30 21:50:57.664186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:57.664147 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" event={"ID":"f60eb6af-5bbd-4a80-b835-60edc3466d68","Type":"ContainerStarted","Data":"bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b"} Oct 30 21:50:57.664657 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:57.664253 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:50:57.776661 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:57.776596 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" podStartSLOduration=1.959653516 podStartE2EDuration="2.776578799s" totalImagesPullingTime="816.925283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:50:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:50:57.775762024 +0000 UTC m=+766.367047511" watchObservedRunningTime="2026-10-30 21:50:57.776578799 +0000 UTC m=+766.367864287" Oct 30 21:50:58.536554 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:58.536511 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:50:58.536720 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:58.536582 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:50:59.686503 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:59.685864 5139 generic.go:356] "Generic (PLEG): container finished" podID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerID="d0091646eb464ed633ffd8710361fc7d45eced785f55eda5876bd9b4c3e896b8" exitCode=0 Oct 30 21:50:59.686503 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:59.685981 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" event={"ID":"dd723b38-6cea-4308-bcf3-26369bbfeecc","Type":"ContainerDied","Data":"d0091646eb464ed633ffd8710361fc7d45eced785f55eda5876bd9b4c3e896b8"} Oct 30 21:50:59.687695 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:59.687166 5139 scope.go:122] "RemoveContainer" containerID="d0091646eb464ed633ffd8710361fc7d45eced785f55eda5876bd9b4c3e896b8" Oct 30 21:50:59.689235 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:59.689183 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:59.689336 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:59.689292 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:50:59.689405 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:50:59.689348 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:00.714947 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:00.714913 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" event={"ID":"dd723b38-6cea-4308-bcf3-26369bbfeecc","Type":"ContainerStarted","Data":"be7974ad0cfe40ea7fac4d396990d64f2b5a1ed5cec82954502dea621a327059"} Oct 30 21:51:01.191280 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:01.190826 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:51:01.191280 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:01.190966 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 21:51:01.191280 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:01.191011 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 21:51:02.213454 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.187370 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 21:51:02.214561 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.214073 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 21:51:02.215552 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.215524 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh"] Oct 30 21:51:02.218586 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.218532 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:02.316838 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.316783 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh"] Oct 30 21:51:02.319524 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.318730 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-4d14e0\"" Oct 30 21:51:02.319524 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.319455 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-4d14e0-dockercfg-vw9vx\"" Oct 30 21:51:02.331353 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.331319 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690d9602-55a9-41f5-8bf0-506dd9e9bcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh\" (UID: \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:02.331549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.331383 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd2gh\" (UniqueName: \"kubernetes.io/projected/690d9602-55a9-41f5-8bf0-506dd9e9bcae-kube-api-access-bd2gh\") pod \"payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh\" (UID: \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:02.432549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.432514 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd2gh\" (UniqueName: \"kubernetes.io/projected/690d9602-55a9-41f5-8bf0-506dd9e9bcae-kube-api-access-bd2gh\") pod \"payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh\" (UID: \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:02.432738 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.432633 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690d9602-55a9-41f5-8bf0-506dd9e9bcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh\" (UID: \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:02.433332 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.433310 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690d9602-55a9-41f5-8bf0-506dd9e9bcae-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh\" (UID: \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:02.523205 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.523104 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd2gh\" (UniqueName: \"kubernetes.io/projected/690d9602-55a9-41f5-8bf0-506dd9e9bcae-kube-api-access-bd2gh\") pod \"payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh\" (UID: \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:02.531859 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.531828 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:02.993436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:02.990515 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh"] Oct 30 21:51:03.190451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:03.187844 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 21:51:03.190451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:03.187908 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 21:51:03.719442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:03.719392 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 21:51:03.739394 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:03.739235 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" event={"ID":"690d9602-55a9-41f5-8bf0-506dd9e9bcae","Type":"ContainerStarted","Data":"39f3b14477c4182b39954121830639fd0ab6afe68e14f0f065828d54a6449237"} Oct 30 21:51:03.739394 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:03.739300 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:03.739394 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:03.739329 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" event={"ID":"690d9602-55a9-41f5-8bf0-506dd9e9bcae","Type":"ContainerStarted","Data":"ff712d31d16acb1e77a7b9f9d00223e4233d1560e72731998856f0ac8ef3ae3b"} Oct 30 21:51:04.188441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:04.187888 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 21:51:04.188441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:04.187955 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 21:51:05.187822 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:05.187782 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 30 21:51:05.188315 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:05.187880 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 30 21:51:06.199281 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:06.199178 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:51:06.200262 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:06.200023 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:51:06.201284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:06.201256 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 21:51:06.408239 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:06.408181 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" podStartSLOduration=4.408167981 podStartE2EDuration="4.408167981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:04.004374283 +0000 UTC m=+772.595659775" watchObservedRunningTime="2026-10-30 21:51:06.408167981 +0000 UTC m=+774.999453511" Oct 30 21:51:08.777455 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:08.776607 5139 generic.go:356] "Generic (PLEG): container finished" podID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerID="7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88" exitCode=0 Oct 30 21:51:08.777455 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:08.776716 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" event={"ID":"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef","Type":"ContainerDied","Data":"7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88"} Oct 30 21:51:08.777455 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:08.777195 5139 scope.go:122] "RemoveContainer" containerID="7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88" Oct 30 21:51:08.790160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:08.790116 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:08.790335 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:08.790202 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:08.790335 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:08.790237 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:09.816654 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:09.816613 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" event={"ID":"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef","Type":"ContainerStarted","Data":"72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d"} Oct 30 21:51:09.849117 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:09.849074 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:51:09.850386 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:09.850364 5139 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:51:09.850461 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:09.850446 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:51:10.849437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:10.849388 5139 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:51:10.849877 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:10.849470 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:51:11.848888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:11.848850 5139 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:51:11.849135 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:11.848916 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:51:12.495156 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:12.495118 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:51:12.852705 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:12.852606 5139 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:51:12.852705 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:12.852680 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:51:13.849228 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:13.849182 5139 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:51:13.849784 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:13.849253 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:51:14.747499 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:14.747462 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 21:51:14.849197 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:14.849164 5139 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 30 21:51:14.849361 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:14.849222 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 30 21:51:15.850072 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:15.849986 5139 generic.go:356] "Generic (PLEG): container finished" podID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerID="5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e" exitCode=0 Oct 30 21:51:15.850552 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:15.850065 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" event={"ID":"9e166eda-5b4a-4d02-9d07-6d577fd9706a","Type":"ContainerDied","Data":"5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e"} Oct 30 21:51:15.850632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:15.850556 5139 scope.go:122] "RemoveContainer" containerID="5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e" Oct 30 21:51:15.851934 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:15.851902 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:15.852019 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:15.851992 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:15.852067 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:15.852045 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:51:15.853794 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:15.853773 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:51:15.855963 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:15.855944 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 21:51:16.856502 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:16.856466 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" event={"ID":"9e166eda-5b4a-4d02-9d07-6d577fd9706a","Type":"ContainerStarted","Data":"4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad"} Oct 30 21:51:17.251148 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:17.251119 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:51:17.251347 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:17.251323 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 21:51:17.251454 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:17.251399 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 21:51:18.251880 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:18.251840 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 21:51:18.252294 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:18.251897 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 21:51:19.251926 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:19.251882 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 21:51:19.252347 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:19.251968 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 21:51:19.812768 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:51:19.812730 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:51:20.251677 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:20.251643 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 21:51:20.251853 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:20.251704 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 21:51:21.251039 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:21.251008 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 21:51:21.251462 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:21.251055 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 21:51:22.256458 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:22.256395 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:51:22.256954 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:22.256852 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:51:22.258619 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:22.258597 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 21:51:24.361775 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.361738 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2"] Oct 30 21:51:24.364826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.364806 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.413740 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.413710 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-3c4802-serving-cert\"" Oct 30 21:51:24.461820 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.461788 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2"] Oct 30 21:51:24.526335 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.526303 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxpgl\" (UniqueName: \"kubernetes.io/projected/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-kube-api-access-sxpgl\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.526519 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.526342 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.526519 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.526458 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-metrics-certs\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.526519 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.526505 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.627974 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.627888 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxpgl\" (UniqueName: \"kubernetes.io/projected/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-kube-api-access-sxpgl\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.627974 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.627929 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.628215 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.627996 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-metrics-certs\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.628215 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.628057 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.628844 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.628818 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.630397 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.630375 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-metrics-certs\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.630518 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.630447 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-maas-api-tls\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.686275 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.686245 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxpgl\" (UniqueName: \"kubernetes.io/projected/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-kube-api-access-sxpgl\") pod \"maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:24.977889 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:24.977806 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:25.345220 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:25.345187 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2"] Oct 30 21:51:25.351805 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:51:25.351779 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaa29a9a_d6e4_4c5c_86f7_bf6bed55faf4.slice/crio-65bf8fb40d61c96719ca89716fa3d53ecefd49463c854a50f8ebe9fb04de6c80 WatchSource:0}: Error finding container 65bf8fb40d61c96719ca89716fa3d53ecefd49463c854a50f8ebe9fb04de6c80: Status 404 returned error can't find the container with id 65bf8fb40d61c96719ca89716fa3d53ecefd49463c854a50f8ebe9fb04de6c80 Oct 30 21:51:25.927384 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:25.927336 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" event={"ID":"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4","Type":"ContainerStarted","Data":"65bf8fb40d61c96719ca89716fa3d53ecefd49463c854a50f8ebe9fb04de6c80"} Oct 30 21:51:26.933640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:26.933599 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" event={"ID":"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4","Type":"ContainerStarted","Data":"cb9ba216219088bee6b569fa31c871bcdfb500fce4e7f67a48ab8eb791e9f0ce"} Oct 30 21:51:26.934030 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:26.933672 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:27.046144 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:27.046087 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" podStartSLOduration=2.142073891 podStartE2EDuration="3.046071616s" totalImagesPullingTime="903.997725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:27.044011545 +0000 UTC m=+795.635297065" watchObservedRunningTime="2026-10-30 21:51:27.046071616 +0000 UTC m=+795.637357103" Oct 30 21:51:32.943845 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:51:32.943814 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 21:51:49.803877 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:51:49.803838 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:52:03.706842 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.706803 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n"] Oct 30 21:52:03.710474 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.710441 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.806159 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.806118 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"default-dockercfg-rz8s4\"" Oct 30 21:52:03.806159 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.806146 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"openshift-service-ca.crt\"" Oct 30 21:52:03.806395 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.806156 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"kube-root-ca.crt\"" Oct 30 21:52:03.806395 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.806258 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"simulator-w2-56f3fe-kserve-self-signed-certs\"" Oct 30 21:52:03.808009 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.807984 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n"] Oct 30 21:52:03.809836 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.809819 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-tmp-dir\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.809895 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.809868 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-home\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.809945 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.809900 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a27a42-dfc1-491d-b955-774c21708323-tls-certs\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.810024 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.810009 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-dshm\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.810066 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.810046 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmkf\" (UniqueName: \"kubernetes.io/projected/a8a27a42-dfc1-491d-b955-774c21708323-kube-api-access-dsmkf\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.810113 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.810079 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-model-cache\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.910734 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.910697 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-tmp-dir\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.910922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.910760 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-home\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.910922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.910796 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a27a42-dfc1-491d-b955-774c21708323-tls-certs\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.910922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.910828 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-dshm\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.910922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.910857 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmkf\" (UniqueName: \"kubernetes.io/projected/a8a27a42-dfc1-491d-b955-774c21708323-kube-api-access-dsmkf\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.910922 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.910892 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-model-cache\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.911225 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.911197 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-tmp-dir\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.911304 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.911274 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-model-cache\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.911455 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.911278 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-home\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.913146 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.913122 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-dshm\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.913400 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.913383 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a27a42-dfc1-491d-b955-774c21708323-tls-certs\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:03.971261 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:03.971186 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmkf\" (UniqueName: \"kubernetes.io/projected/a8a27a42-dfc1-491d-b955-774c21708323-kube-api-access-dsmkf\") pod \"simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:04.022355 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:04.022321 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:04.395013 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:04.394978 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n"] Oct 30 21:52:04.402401 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:52:04.402370 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda8a27a42_dfc1_491d_b955_774c21708323.slice/crio-e6404fee9f4e116cbbeb77ab6f1a0564e6fa19aee17291af2939ffb4d2008a29 WatchSource:0}: Error finding container e6404fee9f4e116cbbeb77ab6f1a0564e6fa19aee17291af2939ffb4d2008a29: Status 404 returned error can't find the container with id e6404fee9f4e116cbbeb77ab6f1a0564e6fa19aee17291af2939ffb4d2008a29 Oct 30 21:52:05.112235 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:05.112186 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" event={"ID":"a8a27a42-dfc1-491d-b955-774c21708323","Type":"ContainerStarted","Data":"1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd"} Oct 30 21:52:05.112235 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:05.112242 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" event={"ID":"a8a27a42-dfc1-491d-b955-774c21708323","Type":"ContainerStarted","Data":"e6404fee9f4e116cbbeb77ab6f1a0564e6fa19aee17291af2939ffb4d2008a29"} Oct 30 21:52:05.225830 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:05.225777 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podStartSLOduration=2.225762382 podStartE2EDuration="2.225762382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:05.223981502 +0000 UTC m=+833.815266990" watchObservedRunningTime="2026-10-30 21:52:05.225762382 +0000 UTC m=+833.817047872" Oct 30 21:52:06.023372 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:06.023341 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:14.023408 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:14.023356 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:14.036454 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:14.036410 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:14.170816 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:14.170785 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:52:19.804364 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:52:19.804318 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:52:23.601217 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.601181 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv"] Oct 30 21:52:23.604240 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.604218 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.699618 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.699578 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"openshift-service-ca.crt\"" Oct 30 21:52:23.699618 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.699578 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"simulator-w3-4d14e0-kserve-self-signed-certs\"" Oct 30 21:52:23.699618 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.699584 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"kube-root-ca.crt\"" Oct 30 21:52:23.699618 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.699590 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"default-dockercfg-7bwxd\"" Oct 30 21:52:23.701393 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.701371 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv"] Oct 30 21:52:23.705089 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.705071 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tls-certs\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.705138 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.705121 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msh99\" (UniqueName: \"kubernetes.io/projected/9249409a-b8c4-4b9b-908a-2402b39d8f4a-kube-api-access-msh99\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.705187 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.705176 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-model-cache\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.705251 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.705236 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-home\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.705356 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.705333 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-dshm\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.705393 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.705364 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tmp-dir\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.805885 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.805851 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-home\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.806081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.805939 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-dshm\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.806081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.805962 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tmp-dir\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.806081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.805987 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tls-certs\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.806081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.806052 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msh99\" (UniqueName: \"kubernetes.io/projected/9249409a-b8c4-4b9b-908a-2402b39d8f4a-kube-api-access-msh99\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.806313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.806084 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-model-cache\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.806313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.806306 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-home\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.806615 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.806595 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-model-cache\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.806702 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.806685 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tmp-dir\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.808362 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.808342 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-dshm\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.808680 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.808657 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tls-certs\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.864051 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.863975 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msh99\" (UniqueName: \"kubernetes.io/projected/9249409a-b8c4-4b9b-908a-2402b39d8f4a-kube-api-access-msh99\") pod \"simulator-w3-4d14e0-kserve-f4c67f658-grdzv\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:23.915810 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:23.915780 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:24.283645 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:24.283615 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv"] Oct 30 21:52:24.289390 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:52:24.289362 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9249409a_b8c4_4b9b_908a_2402b39d8f4a.slice/crio-1bfb28a54249489db1949945b4df055e87859c7ccb9db490fb05acd7129ed327 WatchSource:0}: Error finding container 1bfb28a54249489db1949945b4df055e87859c7ccb9db490fb05acd7129ed327: Status 404 returned error can't find the container with id 1bfb28a54249489db1949945b4df055e87859c7ccb9db490fb05acd7129ed327 Oct 30 21:52:25.216091 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:25.216049 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" event={"ID":"9249409a-b8c4-4b9b-908a-2402b39d8f4a","Type":"ContainerStarted","Data":"df55013654d75b42d1add9ce5b931f01714e6c15a71d7d965163c0f979537575"} Oct 30 21:52:25.216512 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:25.216101 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" event={"ID":"9249409a-b8c4-4b9b-908a-2402b39d8f4a","Type":"ContainerStarted","Data":"1bfb28a54249489db1949945b4df055e87859c7ccb9db490fb05acd7129ed327"} Oct 30 21:52:25.327528 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:25.327475 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podStartSLOduration=2.327460394 podStartE2EDuration="2.327460394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:25.326429797 +0000 UTC m=+853.917715278" watchObservedRunningTime="2026-10-30 21:52:25.327460394 +0000 UTC m=+853.918745881" Oct 30 21:52:25.916429 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:25.916389 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:33.916212 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:33.916114 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:33.928737 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:33.928711 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:34.271519 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:34.271483 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:52:43.774171 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.774134 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl"] Oct 30 21:52:43.777642 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.777604 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:43.827234 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.827210 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"openshift-service-ca.crt\"" Oct 30 21:52:43.827234 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.827212 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"kube-root-ca.crt\"" Oct 30 21:52:43.873640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.873604 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"premium-w4-26743a-kserve-self-signed-certs\"" Oct 30 21:52:43.873842 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.873663 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"default-dockercfg-c758n\"" Oct 30 21:52:43.875538 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.875515 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl"] Oct 30 21:52:43.911012 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.910980 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt6lb\" (UniqueName: \"kubernetes.io/projected/90a132e5-9eef-4f67-a14f-5d74e05f4e75-kube-api-access-nt6lb\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:43.911249 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.911044 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-model-cache\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:43.911249 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.911132 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-dshm\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:43.911249 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.911184 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-home\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:43.911386 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.911269 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tmp-dir\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:43.911386 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:43.911319 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tls-certs\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012061 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012025 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt6lb\" (UniqueName: \"kubernetes.io/projected/90a132e5-9eef-4f67-a14f-5d74e05f4e75-kube-api-access-nt6lb\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012259 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012089 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-model-cache\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012259 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012107 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-dshm\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012259 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012124 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-home\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012259 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012158 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tmp-dir\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012259 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012184 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tls-certs\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012607 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012583 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-home\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012607 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012597 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tmp-dir\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.012709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.012643 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-model-cache\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.014585 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.014562 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-dshm\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.014747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.014732 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tls-certs\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.073161 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.073097 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt6lb\" (UniqueName: \"kubernetes.io/projected/90a132e5-9eef-4f67-a14f-5d74e05f4e75-kube-api-access-nt6lb\") pod \"premium-w4-26743a-kserve-7b8c7bb9b7-92vtl\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.089006 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.088981 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:44.455744 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:44.455659 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl"] Oct 30 21:52:44.461434 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:52:44.461389 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90a132e5_9eef_4f67_a14f_5d74e05f4e75.slice/crio-b2b57a4d2a6d16eab9f9c54af3873d4a682a21cc4de0fe890afa3dc267ae43bf WatchSource:0}: Error finding container b2b57a4d2a6d16eab9f9c54af3873d4a682a21cc4de0fe890afa3dc267ae43bf: Status 404 returned error can't find the container with id b2b57a4d2a6d16eab9f9c54af3873d4a682a21cc4de0fe890afa3dc267ae43bf Oct 30 21:52:45.314898 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:45.314862 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" event={"ID":"90a132e5-9eef-4f67-a14f-5d74e05f4e75","Type":"ContainerStarted","Data":"dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da"} Oct 30 21:52:45.315278 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:45.314908 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" event={"ID":"90a132e5-9eef-4f67-a14f-5d74e05f4e75","Type":"ContainerStarted","Data":"b2b57a4d2a6d16eab9f9c54af3873d4a682a21cc4de0fe890afa3dc267ae43bf"} Oct 30 21:52:45.428904 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:45.428839 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podStartSLOduration=2.428806028 podStartE2EDuration="2.428806028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:45.428526252 +0000 UTC m=+874.019811738" watchObservedRunningTime="2026-10-30 21:52:45.428806028 +0000 UTC m=+874.020091515" Oct 30 21:52:46.090020 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:46.089978 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:49.814385 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:52:49.814343 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:52:54.090001 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:54.089962 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:54.102572 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:54.102546 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:52:54.367637 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:52:54.367562 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:53:19.803920 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:53:19.803885 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:53:28.746647 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.746569 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht"] Oct 30 21:53:28.749672 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.749649 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.829456 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.829429 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"premium-w3-4d14e0-kserve-self-signed-certs\"" Oct 30 21:53:28.847329 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.847299 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht"] Oct 30 21:53:28.871139 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.871110 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9092c-a737-48f9-99c4-675c8d0da106-tls-certs\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.871279 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.871148 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-model-cache\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.871279 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.871256 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-dshm\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.871354 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.871319 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-tmp-dir\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.871389 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.871374 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-home\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.871458 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.871406 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvxxr\" (UniqueName: \"kubernetes.io/projected/aba9092c-a737-48f9-99c4-675c8d0da106-kube-api-access-dvxxr\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973011 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.972977 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9092c-a737-48f9-99c4-675c8d0da106-tls-certs\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973011 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.973018 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-model-cache\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973231 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.973073 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-dshm\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973231 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.973141 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-tmp-dir\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973231 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.973183 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-home\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973390 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.973250 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvxxr\" (UniqueName: \"kubernetes.io/projected/aba9092c-a737-48f9-99c4-675c8d0da106-kube-api-access-dvxxr\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973601 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.973576 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-model-cache\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973689 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.973647 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-tmp-dir\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.973738 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.973688 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-home\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.975535 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.975512 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-dshm\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:28.975634 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:28.975523 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9092c-a737-48f9-99c4-675c8d0da106-tls-certs\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:29.032350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:29.032278 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvxxr\" (UniqueName: \"kubernetes.io/projected/aba9092c-a737-48f9-99c4-675c8d0da106-kube-api-access-dvxxr\") pod \"premium-w3-4d14e0-kserve-7c5b98d799-qdhht\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:29.060756 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:29.060724 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:29.461114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:29.461031 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht"] Oct 30 21:53:29.467570 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:53:29.467546 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaba9092c_a737_48f9_99c4_675c8d0da106.slice/crio-5b0547d36bec79315b53f855562fef7bf080e864162e19ec70c6e02d10994cf2 WatchSource:0}: Error finding container 5b0547d36bec79315b53f855562fef7bf080e864162e19ec70c6e02d10994cf2: Status 404 returned error can't find the container with id 5b0547d36bec79315b53f855562fef7bf080e864162e19ec70c6e02d10994cf2 Oct 30 21:53:29.534826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:29.534789 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" event={"ID":"aba9092c-a737-48f9-99c4-675c8d0da106","Type":"ContainerStarted","Data":"5b0547d36bec79315b53f855562fef7bf080e864162e19ec70c6e02d10994cf2"} Oct 30 21:53:30.541355 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:30.541319 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" event={"ID":"aba9092c-a737-48f9-99c4-675c8d0da106","Type":"ContainerStarted","Data":"4cb7ff5d69dec4b604abf0017d7b1f120fd3f166523405d8566b5daff24eabc7"} Oct 30 21:53:30.653676 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:30.653624 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podStartSLOduration=2.653610408 podStartE2EDuration="2.653610408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:30.65283028 +0000 UTC m=+919.244115749" watchObservedRunningTime="2026-10-30 21:53:30.653610408 +0000 UTC m=+919.244895895" Oct 30 21:53:31.061038 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:31.061004 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:39.061644 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:39.061605 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:39.074534 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:39.074494 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:39.599653 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:53:39.599624 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:53:49.804292 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:53:49.804253 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:54:19.804603 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:54:19.804565 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:54:49.805801 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:54:49.805753 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:54:55.104577 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.104486 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6"] Oct 30 21:54:55.109107 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.109080 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.160014 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.159979 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3c4802-openshift-default-dockercfg-6gcqv\"" Oct 30 21:54:55.162744 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.162700 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snv8l\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-kube-api-access-snv8l\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.162913 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.162757 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.162913 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.162895 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.163058 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.162945 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.163058 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.162970 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.163058 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.163011 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.163248 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.163101 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.163248 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.163126 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.163248 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.163181 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27903e11-7864-4b21-84a8-17ed8c9ca170-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.209572 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.209538 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6"] Oct 30 21:54:55.264263 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.264231 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.264511 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.264312 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.264511 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.264329 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.264653 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.264587 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27903e11-7864-4b21-84a8-17ed8c9ca170-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.264714 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.264667 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snv8l\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-kube-api-access-snv8l\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.264714 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.264708 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.264846 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.264771 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.264905 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.264848 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.265090 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.265062 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.265393 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.265365 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.265617 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.265390 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27903e11-7864-4b21-84a8-17ed8c9ca170-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.265742 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.265674 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.265742 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.265708 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.265969 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.265941 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.267648 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.267620 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.268193 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.268170 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.324259 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.324218 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snv8l\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-kube-api-access-snv8l\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.324259 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.324243 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.423282 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.423198 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:55.852607 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.852541 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6"] Oct 30 21:54:55.855180 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:54:55.855139 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27903e11_7864_4b21_84a8_17ed8c9ca170.slice/crio-0134883edcaab2cd4bfcce3921d967aa04b260986171908e032d39756930c2a8 WatchSource:0}: Error finding container 0134883edcaab2cd4bfcce3921d967aa04b260986171908e032d39756930c2a8: Status 404 returned error can't find the container with id 0134883edcaab2cd4bfcce3921d967aa04b260986171908e032d39756930c2a8 Oct 30 21:54:55.857574 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.857535 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:55.857686 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.857619 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:55.857686 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.857660 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:54:55.976753 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.976710 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" event={"ID":"27903e11-7864-4b21-84a8-17ed8c9ca170","Type":"ContainerStarted","Data":"aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452"} Oct 30 21:54:55.976920 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:55.976775 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" event={"ID":"27903e11-7864-4b21-84a8-17ed8c9ca170","Type":"ContainerStarted","Data":"0134883edcaab2cd4bfcce3921d967aa04b260986171908e032d39756930c2a8"} Oct 30 21:54:56.093900 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:56.093845 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podStartSLOduration=1.093830558 podStartE2EDuration="1.093830558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:56.091781933 +0000 UTC m=+1004.683067420" watchObservedRunningTime="2026-10-30 21:54:56.093830558 +0000 UTC m=+1004.685116045" Oct 30 21:54:56.423948 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:56.423920 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:54:56.425791 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:56.425760 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:54:56.425960 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:56.425810 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:54:57.423938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:57.423901 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:54:57.424119 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:57.423954 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:54:58.423731 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:58.423698 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:54:58.423911 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:58.423752 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:54:59.424329 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:59.424292 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:54:59.424766 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:54:59.424349 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:55:00.424169 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:00.424130 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:55:00.424343 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:00.424196 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:55:01.424182 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:01.424144 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:55:01.424641 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:01.424191 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:55:02.423631 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:02.423594 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Oct 30 21:55:02.423809 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:02.423656 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Oct 30 21:55:03.428367 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:03.428335 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:55:03.428826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:03.428641 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:55:03.429463 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:03.429439 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 21:55:11.682483 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:11.682456 5139 scope.go:122] "RemoveContainer" containerID="98b3a7b3f05f9e146b75a7b03e91fcd1a85bfc40339292999b2954c5e1c8a435" Oct 30 21:55:13.416261 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:13.416215 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:55:13.416690 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:13.416293 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:55:15.050658 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.050614 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6"] Oct 30 21:55:15.055805 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.055774 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.168823 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.168776 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6"] Oct 30 21:55:15.186283 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186253 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.186502 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186290 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.186502 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186457 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbpn4\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-kube-api-access-sbpn4\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.186680 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186509 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.186680 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186576 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.186680 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186628 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.186818 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186681 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.186870 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186826 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.186913 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.186871 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288198 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288160 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288198 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288204 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288252 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288299 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288354 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288376 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288471 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288504 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288577 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbpn4\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-kube-api-access-sbpn4\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288680 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.288888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288778 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.289070 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288914 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.289070 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.288936 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.289070 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.289012 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.290953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.290928 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.291052 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.291028 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.346243 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.346165 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.347322 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.347300 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbpn4\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-kube-api-access-sbpn4\") pod \"maas-default-gateway-openshift-default-7df854bf67-6ljr6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.369202 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.369176 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:15.786703 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.786678 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6"] Oct 30 21:55:15.788298 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:55:15.788267 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a114fa_b7ff_4186_ba9e_0a5d0ba29ce6.slice/crio-e12a2341eff369691dc04c6866700473c90122a75a5487e41fbb9a9786d3c9e2 WatchSource:0}: Error finding container e12a2341eff369691dc04c6866700473c90122a75a5487e41fbb9a9786d3c9e2: Status 404 returned error can't find the container with id e12a2341eff369691dc04c6866700473c90122a75a5487e41fbb9a9786d3c9e2 Oct 30 21:55:15.791009 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.790975 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:15.791088 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.791039 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:15.791088 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:15.791072 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:16.086478 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:16.086351 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" event={"ID":"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6","Type":"ContainerStarted","Data":"224d7dc9f714aca7ba52569bdaf5ad5880cb2346a424965be0eef8a6bbab86d7"} Oct 30 21:55:16.086478 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:16.086463 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" event={"ID":"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6","Type":"ContainerStarted","Data":"e12a2341eff369691dc04c6866700473c90122a75a5487e41fbb9a9786d3c9e2"} Oct 30 21:55:16.202354 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:16.202301 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podStartSLOduration=1.2022867 podStartE2EDuration="1.2022867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:16.200183903 +0000 UTC m=+1024.791469391" watchObservedRunningTime="2026-10-30 21:55:16.2022867 +0000 UTC m=+1024.793572186" Oct 30 21:55:16.371499 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:16.369781 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:16.386599 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:16.386552 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:55:16.386816 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:16.386625 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:55:17.383440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:17.378594 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:55:17.383440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:17.378655 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:55:18.379441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:18.377880 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:55:18.379441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:18.377946 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:55:19.385441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:19.383836 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:55:19.385441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:19.383901 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:55:19.814004 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:55:19.813965 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:55:20.379442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:20.370896 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:55:20.379442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:20.370960 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:55:21.384442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:21.381812 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:55:21.384442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:21.381878 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:55:22.370268 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:22.370228 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:55:22.370468 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:22.370288 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:55:23.369599 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:23.369561 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Oct 30 21:55:23.370033 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:23.369638 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Oct 30 21:55:24.555723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:24.555670 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:24.557078 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:24.557054 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:24.557717 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:24.557698 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 21:55:30.033011 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.032975 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5"] Oct 30 21:55:30.036762 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.036740 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083224 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263d7b9-e982-4966-a364-09b01ea59e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083354 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263d7b9-e982-4966-a364-09b01ea59e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083410 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083529 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083556 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083590 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083633 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083722 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.088467 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.083753 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xtc\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-kube-api-access-78xtc\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185069 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185103 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185110 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xtc\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-kube-api-access-78xtc\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185355 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185235 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263d7b9-e982-4966-a364-09b01ea59e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185355 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185339 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263d7b9-e982-4966-a364-09b01ea59e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185522 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185392 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185522 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185508 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185630 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185539 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185630 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185567 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185630 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185605 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185766 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185510 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185804 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185760 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185873 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185845 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.185951 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.185934 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.186050 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.186021 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263d7b9-e982-4966-a364-09b01ea59e50-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.187837 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.187813 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.187931 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.187888 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263d7b9-e982-4966-a364-09b01ea59e50-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.241583 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.241549 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5"] Oct 30 21:55:30.246388 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.246357 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xtc\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-kube-api-access-78xtc\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.248113 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.248083 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-lpfj5\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.349563 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.349472 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:30.769388 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.769350 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5"] Oct 30 21:55:30.801347 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:55:30.801316 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode263d7b9_e982_4966_a364_09b01ea59e50.slice/crio-a2490c0e9bac728d010edb0ba987a4775d9a4a280f04b13dd0431339c9e91e07 WatchSource:0}: Error finding container a2490c0e9bac728d010edb0ba987a4775d9a4a280f04b13dd0431339c9e91e07: Status 404 returned error can't find the container with id a2490c0e9bac728d010edb0ba987a4775d9a4a280f04b13dd0431339c9e91e07 Oct 30 21:55:30.803793 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.803756 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:30.803886 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.803831 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:30.803886 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:30.803861 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:55:31.165731 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:31.165652 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" event={"ID":"e263d7b9-e982-4966-a364-09b01ea59e50","Type":"ContainerStarted","Data":"39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b"} Oct 30 21:55:31.165731 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:31.165712 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" event={"ID":"e263d7b9-e982-4966-a364-09b01ea59e50","Type":"ContainerStarted","Data":"a2490c0e9bac728d010edb0ba987a4775d9a4a280f04b13dd0431339c9e91e07"} Oct 30 21:55:31.328176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:31.328116 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podStartSLOduration=2.328103854 podStartE2EDuration="2.328103854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:31.325803925 +0000 UTC m=+1039.917089412" watchObservedRunningTime="2026-10-30 21:55:31.328103854 +0000 UTC m=+1039.919389342" Oct 30 21:55:31.349898 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:31.349867 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:31.351471 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:31.351443 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:55:31.351611 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:31.351496 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:55:32.350060 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:32.350029 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:55:32.350513 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:32.350079 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:55:33.350102 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:33.350063 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:55:33.350542 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:33.350121 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:55:34.350615 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:34.350574 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:55:34.351105 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:34.350640 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:55:35.349995 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:35.349958 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:55:35.350182 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:35.350019 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:55:36.350879 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:36.350836 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:55:36.351371 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:36.350900 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:55:37.350233 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:37.350196 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:55:37.350387 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:37.350253 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:55:38.350133 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:38.350091 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Oct 30 21:55:38.350512 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:38.350200 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Oct 30 21:55:39.546153 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:39.546119 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:39.546683 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:39.546533 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:39.547706 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:39.547684 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 21:55:49.804882 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:55:49.804838 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:55:57.897350 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.897310 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq"] Oct 30 21:55:57.900672 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.900651 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:57.929447 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.929400 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-dshm\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:57.929586 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.929470 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-home\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:57.929728 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.929628 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m84hh\" (UniqueName: \"kubernetes.io/projected/ede50429-a542-4b26-8921-7226330d50e6-kube-api-access-m84hh\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:57.929825 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.929796 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-model-cache\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:57.929885 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.929873 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede50429-a542-4b26-8921-7226330d50e6-tls-certs\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:57.929941 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.929911 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-tmp-dir\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:57.996498 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.996463 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"kube-root-ca.crt\"" Oct 30 21:55:57.996498 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.996495 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"openshift-service-ca.crt\"" Oct 30 21:55:57.996744 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.996507 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"distinct-w0-9a5623-kserve-self-signed-certs\"" Oct 30 21:55:57.996744 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.996495 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"default-dockercfg-45vff\"" Oct 30 21:55:57.998519 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:57.998499 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq"] Oct 30 21:55:58.031161 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031136 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-model-cache\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.031301 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031193 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede50429-a542-4b26-8921-7226330d50e6-tls-certs\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.031373 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031351 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-tmp-dir\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.031484 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031466 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-dshm\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.031545 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031518 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-home\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.031613 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031547 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-model-cache\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.031656 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031611 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m84hh\" (UniqueName: \"kubernetes.io/projected/ede50429-a542-4b26-8921-7226330d50e6-kube-api-access-m84hh\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.031718 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031695 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-tmp-dir\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.031892 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.031829 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-home\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.033570 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.033549 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-dshm\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.033747 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.033730 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede50429-a542-4b26-8921-7226330d50e6-tls-certs\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.089954 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.089921 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m84hh\" (UniqueName: \"kubernetes.io/projected/ede50429-a542-4b26-8921-7226330d50e6-kube-api-access-m84hh\") pod \"distinct-w0-9a5623-kserve-5478f97cff-rdptq\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.213003 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.212914 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:55:58.579589 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:58.579563 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq"] Oct 30 21:55:58.581553 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:55:58.581524 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podede50429_a542_4b26_8921_7226330d50e6.slice/crio-9c3333acd170b413b5cdea8b4aa9b770b70b8ab6b5398b03c84a4165473d0627 WatchSource:0}: Error finding container 9c3333acd170b413b5cdea8b4aa9b770b70b8ab6b5398b03c84a4165473d0627: Status 404 returned error can't find the container with id 9c3333acd170b413b5cdea8b4aa9b770b70b8ab6b5398b03c84a4165473d0627 Oct 30 21:55:59.331472 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:59.331438 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" event={"ID":"ede50429-a542-4b26-8921-7226330d50e6","Type":"ContainerStarted","Data":"7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c"} Oct 30 21:55:59.331472 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:59.331480 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" event={"ID":"ede50429-a542-4b26-8921-7226330d50e6","Type":"ContainerStarted","Data":"9c3333acd170b413b5cdea8b4aa9b770b70b8ab6b5398b03c84a4165473d0627"} Oct 30 21:55:59.443509 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:55:59.443446 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podStartSLOduration=2.4434313899999998 podStartE2EDuration="2.44343139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:59.441639845 +0000 UTC m=+1068.032925333" watchObservedRunningTime="2026-10-30 21:55:59.44343139 +0000 UTC m=+1068.034716871" Oct 30 21:56:00.213671 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:00.213631 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:56:08.213495 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:08.213441 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:56:08.226539 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:08.226511 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:56:08.390972 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:08.390942 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:56:19.806062 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:56:19.806017 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:56:29.799673 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.799585 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44"] Oct 30 21:56:29.803977 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.803952 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.858797 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.858764 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44"] Oct 30 21:56:29.890980 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.890947 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-home\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.890980 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.890981 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-model-cache\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.891176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.891061 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7510e0d7-368a-4310-8274-f661eb93a6e0-tls-certs\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.891176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.891114 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-dshm\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.891176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.891152 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cghv5\" (UniqueName: \"kubernetes.io/projected/7510e0d7-368a-4310-8274-f661eb93a6e0-kube-api-access-cghv5\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.891388 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.891365 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-tmp-dir\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.902647 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.902623 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"default-dockercfg-7bt4g\"" Oct 30 21:56:29.902647 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.902626 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"openshift-service-ca.crt\"" Oct 30 21:56:29.902812 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.902670 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"kube-root-ca.crt\"" Oct 30 21:56:29.902812 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.902715 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"e2e-conflict-b-09506444-kserve-self-signed-certs\"" Oct 30 21:56:29.992297 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992250 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-tmp-dir\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.992536 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992384 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-home\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.992536 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992432 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-model-cache\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.992536 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992494 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7510e0d7-368a-4310-8274-f661eb93a6e0-tls-certs\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.992716 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992560 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-dshm\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.992716 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992639 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cghv5\" (UniqueName: \"kubernetes.io/projected/7510e0d7-368a-4310-8274-f661eb93a6e0-kube-api-access-cghv5\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.992716 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992707 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-tmp-dir\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.992880 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992756 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-home\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.992942 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.992901 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-model-cache\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.995183 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.995155 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7510e0d7-368a-4310-8274-f661eb93a6e0-tls-certs\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:29.995324 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:29.995276 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-dshm\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:30.051604 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:30.051528 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cghv5\" (UniqueName: \"kubernetes.io/projected/7510e0d7-368a-4310-8274-f661eb93a6e0-kube-api-access-cghv5\") pod \"e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:30.116427 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:30.116378 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:30.505977 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:30.505950 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44"] Oct 30 21:56:30.507851 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:56:30.507813 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7510e0d7_368a_4310_8274_f661eb93a6e0.slice/crio-2602f83bfa62b90674f063ab779f93fcf1e48acae5e219239a7c5a33637924f5 WatchSource:0}: Error finding container 2602f83bfa62b90674f063ab779f93fcf1e48acae5e219239a7c5a33637924f5: Status 404 returned error can't find the container with id 2602f83bfa62b90674f063ab779f93fcf1e48acae5e219239a7c5a33637924f5 Oct 30 21:56:31.502767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:31.502724 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" event={"ID":"7510e0d7-368a-4310-8274-f661eb93a6e0","Type":"ContainerStarted","Data":"8c8ae9af399ab586fb60efa84255fdd1023905476d61599c99643a839e1935be"} Oct 30 21:56:31.502767 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:31.502766 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" event={"ID":"7510e0d7-368a-4310-8274-f661eb93a6e0","Type":"ContainerStarted","Data":"2602f83bfa62b90674f063ab779f93fcf1e48acae5e219239a7c5a33637924f5"} Oct 30 21:56:31.615070 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:31.615021 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" podStartSLOduration=2.615006713 podStartE2EDuration="2.615006713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:31.612707892 +0000 UTC m=+1100.203993381" watchObservedRunningTime="2026-10-30 21:56:31.615006713 +0000 UTC m=+1100.206292201" Oct 30 21:56:32.117373 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:32.117342 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:40.121264 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:40.121228 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:40.139782 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:40.139749 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:40.577695 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:56:40.577665 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:56:49.804269 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:56:49.804225 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:57:19.804299 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:57:19.804259 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:57:22.896605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.896567 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f"] Oct 30 21:57:22.901898 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.901877 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:22.986981 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.986947 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-home\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:22.987150 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.987005 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2dx5\" (UniqueName: \"kubernetes.io/projected/309f1902-d2a6-4c31-9ace-958449a06cde-kube-api-access-v2dx5\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:22.987210 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.987146 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-tmp-dir\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:22.987210 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.987196 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-dshm\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:22.987286 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.987277 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309f1902-d2a6-4c31-9ace-958449a06cde-tls-certs\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:22.987322 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.987301 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-model-cache\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:22.999324 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:22.999303 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"distinct-2-w0-9a5623-kserve-self-signed-certs\"" Oct 30 21:57:23.001312 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.001293 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f"] Oct 30 21:57:23.088789 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.088755 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-dshm\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.088992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.088815 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309f1902-d2a6-4c31-9ace-958449a06cde-tls-certs\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.088992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.088859 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-model-cache\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.088992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.088936 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-home\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.088992 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.088968 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2dx5\" (UniqueName: \"kubernetes.io/projected/309f1902-d2a6-4c31-9ace-958449a06cde-kube-api-access-v2dx5\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.089213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.089093 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-tmp-dir\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.089361 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.089329 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-home\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.089361 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.089350 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-model-cache\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.089581 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.089396 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-tmp-dir\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.091221 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.091202 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-dshm\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.091530 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.091510 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309f1902-d2a6-4c31-9ace-958449a06cde-tls-certs\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.150053 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.149974 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2dx5\" (UniqueName: \"kubernetes.io/projected/309f1902-d2a6-4c31-9ace-958449a06cde-kube-api-access-v2dx5\") pod \"distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.213117 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.213059 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:23.581500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.581476 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f"] Oct 30 21:57:23.584328 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:57:23.584298 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod309f1902_d2a6_4c31_9ace_958449a06cde.slice/crio-2d55a4e21e34888e9d7426c069d37b43d734913823f4afdb40c1d470f63a8c22 WatchSource:0}: Error finding container 2d55a4e21e34888e9d7426c069d37b43d734913823f4afdb40c1d470f63a8c22: Status 404 returned error can't find the container with id 2d55a4e21e34888e9d7426c069d37b43d734913823f4afdb40c1d470f63a8c22 Oct 30 21:57:23.799554 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.799513 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" event={"ID":"309f1902-d2a6-4c31-9ace-958449a06cde","Type":"ContainerStarted","Data":"8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741"} Oct 30 21:57:23.799719 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.799563 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" event={"ID":"309f1902-d2a6-4c31-9ace-958449a06cde","Type":"ContainerStarted","Data":"2d55a4e21e34888e9d7426c069d37b43d734913823f4afdb40c1d470f63a8c22"} Oct 30 21:57:23.911040 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:23.910939 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podStartSLOduration=1.910925564 podStartE2EDuration="1.910925564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:23.909046882 +0000 UTC m=+1152.500332370" watchObservedRunningTime="2026-10-30 21:57:23.910925564 +0000 UTC m=+1152.502211051" Oct 30 21:57:24.214114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:24.214029 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:33.214051 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:33.214013 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:33.226436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:33.226387 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:33.870512 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:33.870424 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:57:38.690052 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:38.690017 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44"] Oct 30 21:57:49.803980 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:57:49.803944 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:57:53.710071 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:53.709984 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" podUID="7510e0d7-368a-4310-8274-f661eb93a6e0" containerName="main" containerID="cri-o://8c8ae9af399ab586fb60efa84255fdd1023905476d61599c99643a839e1935be" gracePeriod=45 Oct 30 21:57:53.965196 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:53.965128 5139 generic.go:356] "Generic (PLEG): container finished" podID="7510e0d7-368a-4310-8274-f661eb93a6e0" containerID="8c8ae9af399ab586fb60efa84255fdd1023905476d61599c99643a839e1935be" exitCode=0 Oct 30 21:57:53.965196 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:53.965168 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" event={"ID":"7510e0d7-368a-4310-8274-f661eb93a6e0","Type":"ContainerDied","Data":"8c8ae9af399ab586fb60efa84255fdd1023905476d61599c99643a839e1935be"} Oct 30 21:57:53.965196 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:53.965195 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" event={"ID":"7510e0d7-368a-4310-8274-f661eb93a6e0","Type":"ContainerDied","Data":"2602f83bfa62b90674f063ab779f93fcf1e48acae5e219239a7c5a33637924f5"} Oct 30 21:57:53.965454 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:53.965208 5139 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2602f83bfa62b90674f063ab779f93fcf1e48acae5e219239a7c5a33637924f5" Oct 30 21:57:53.972047 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:53.972024 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:57:54.025594 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.025566 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-home\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-home\") pod \"7510e0d7-368a-4310-8274-f661eb93a6e0\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " Oct 30 21:57:54.025784 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.025606 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-model-cache\") pod \"7510e0d7-368a-4310-8274-f661eb93a6e0\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " Oct 30 21:57:54.025784 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.025639 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7510e0d7-368a-4310-8274-f661eb93a6e0-kube-api-access-cghv5\" (UniqueName: \"kubernetes.io/projected/7510e0d7-368a-4310-8274-f661eb93a6e0-kube-api-access-cghv5\") pod \"7510e0d7-368a-4310-8274-f661eb93a6e0\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " Oct 30 21:57:54.025784 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.025664 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-tmp-dir\") pod \"7510e0d7-368a-4310-8274-f661eb93a6e0\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " Oct 30 21:57:54.025949 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.025797 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-dshm\") pod \"7510e0d7-368a-4310-8274-f661eb93a6e0\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " Oct 30 21:57:54.025949 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.025861 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7510e0d7-368a-4310-8274-f661eb93a6e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/7510e0d7-368a-4310-8274-f661eb93a6e0-tls-certs\") pod \"7510e0d7-368a-4310-8274-f661eb93a6e0\" (UID: \"7510e0d7-368a-4310-8274-f661eb93a6e0\") " Oct 30 21:57:54.025949 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.025847 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-home" pod "7510e0d7-368a-4310-8274-f661eb93a6e0" (UID: "7510e0d7-368a-4310-8274-f661eb93a6e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.025949 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.025930 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-model-cache" pod "7510e0d7-368a-4310-8274-f661eb93a6e0" (UID: "7510e0d7-368a-4310-8274-f661eb93a6e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.026145 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.026090 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-tmp-dir" pod "7510e0d7-368a-4310-8274-f661eb93a6e0" (UID: "7510e0d7-368a-4310-8274-f661eb93a6e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.026476 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.026441 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.026476 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.026469 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.026636 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.026487 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.028092 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.028065 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-dshm" pod "7510e0d7-368a-4310-8274-f661eb93a6e0" (UID: "7510e0d7-368a-4310-8274-f661eb93a6e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 21:57:54.028189 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.028091 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7510e0d7-368a-4310-8274-f661eb93a6e0-tls-certs" pod "7510e0d7-368a-4310-8274-f661eb93a6e0" (UID: "7510e0d7-368a-4310-8274-f661eb93a6e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 21:57:54.028189 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.028169 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7510e0d7-368a-4310-8274-f661eb93a6e0-kube-api-access-cghv5" pod "7510e0d7-368a-4310-8274-f661eb93a6e0" (UID: "7510e0d7-368a-4310-8274-f661eb93a6e0"). InnerVolumeSpecName "kube-api-access-cghv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 21:57:54.127114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.127078 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cghv5\" (UniqueName: \"kubernetes.io/projected/7510e0d7-368a-4310-8274-f661eb93a6e0-kube-api-access-cghv5\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.127114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.127107 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7510e0d7-368a-4310-8274-f661eb93a6e0-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.127114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.127117 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7510e0d7-368a-4310-8274-f661eb93a6e0-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 21:57:54.700128 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.700095 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p"] Oct 30 21:57:54.700585 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.700571 5139 cpu_manager.go:414] "containerMap: removing container" podUID="7510e0d7-368a-4310-8274-f661eb93a6e0" containerName="main" Oct 30 21:57:54.700636 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.700586 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7510e0d7-368a-4310-8274-f661eb93a6e0" containerName="main" Oct 30 21:57:54.700684 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.700674 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="7510e0d7-368a-4310-8274-f661eb93a6e0" containerName="main" Oct 30 21:57:54.704108 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.704090 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733255 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733217 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d6pz\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-kube-api-access-8d6pz\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733671 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733382 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2ab003-110a-40b8-b0bc-a86f7f334772-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733671 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733432 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733671 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733498 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733671 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733559 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733671 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733619 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733671 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733660 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733947 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733705 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.733947 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.733731 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.753122 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.753093 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4d14e0-openshift-default-dockercfg-mqz9f\"" Oct 30 21:57:54.802276 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.802245 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p"] Oct 30 21:57:54.834828 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.834795 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2ab003-110a-40b8-b0bc-a86f7f334772-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.834998 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.834832 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.834998 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.834860 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.834998 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.834902 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.834998 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.834929 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.834998 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.834966 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.835254 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.834996 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.835254 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.835038 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.835254 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.835111 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d6pz\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-kube-api-access-8d6pz\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.835254 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.835223 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.835502 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.835481 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.835694 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.835662 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.835766 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.835706 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2ab003-110a-40b8-b0bc-a86f7f334772-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.835766 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.835724 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.837789 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.837763 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.838125 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.838101 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.892485 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.892456 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.892794 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.892776 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d6pz\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-kube-api-access-8d6pz\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:54.970912 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:54.970839 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44" Oct 30 21:57:54.992652 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:57:54.992620 5139 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7510e0d7_368a_4310_8274_f661eb93a6e0.slice\": RecentStats: unable to find data in memory cache]" Oct 30 21:57:55.022927 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.022898 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:55.124597 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.124566 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44"] Oct 30 21:57:55.132442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.132408 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-b-09506444-kserve-79c6b6dcb5-fll44"] Oct 30 21:57:55.439278 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.439254 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p"] Oct 30 21:57:55.440845 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:57:55.440814 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2ab003_110a_40b8_b0bc_a86f7f334772.slice/crio-b9aa30e72f5e4e9eec66bb9395ae8b912d0b4111830b1dfadfa7a3475ed48954 WatchSource:0}: Error finding container b9aa30e72f5e4e9eec66bb9395ae8b912d0b4111830b1dfadfa7a3475ed48954: Status 404 returned error can't find the container with id b9aa30e72f5e4e9eec66bb9395ae8b912d0b4111830b1dfadfa7a3475ed48954 Oct 30 21:57:55.443253 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.443215 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:57:55.443337 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.443286 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:57:55.443337 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.443314 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:57:55.544880 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.544839 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7510e0d7-368a-4310-8274-f661eb93a6e0" path="/var/lib/kubelet/pods/7510e0d7-368a-4310-8274-f661eb93a6e0/volumes" Oct 30 21:57:55.990555 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.990516 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" event={"ID":"ae2ab003-110a-40b8-b0bc-a86f7f334772","Type":"ContainerStarted","Data":"0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe"} Oct 30 21:57:55.991012 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:55.990588 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" event={"ID":"ae2ab003-110a-40b8-b0bc-a86f7f334772","Type":"ContainerStarted","Data":"b9aa30e72f5e4e9eec66bb9395ae8b912d0b4111830b1dfadfa7a3475ed48954"} Oct 30 21:57:56.023797 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:56.023766 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:57:56.025697 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:56.025667 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:57:56.025830 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:56.025727 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:57:56.104506 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:56.104453 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podStartSLOduration=2.104408113 podStartE2EDuration="2.104408113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:56.103159207 +0000 UTC m=+1184.694444692" watchObservedRunningTime="2026-10-30 21:57:56.104408113 +0000 UTC m=+1184.695693598" Oct 30 21:57:57.023597 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:57.023559 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:57:57.024006 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:57.023625 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:57:58.023493 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:58.023454 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:57:58.023675 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:58.023513 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:57:59.023584 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:59.023550 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:57:59.023751 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:57:59.023600 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:58:00.023592 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.023554 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:58:00.023790 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.023622 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:58:00.845791 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.845758 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc"] Oct 30 21:58:00.849496 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.849471 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:00.924910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.924875 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a377639d-cd1a-4182-8067-fb2071641be1-tls-certs\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:00.924910 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.924919 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-home\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:00.925131 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.925047 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-model-cache\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:00.925178 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.925146 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lhrg\" (UniqueName: \"kubernetes.io/projected/a377639d-cd1a-4182-8067-fb2071641be1-kube-api-access-5lhrg\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:00.925232 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.925218 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-dshm\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:00.925321 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.925308 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-tmp-dir\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:00.945007 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.944982 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"distinct-2-w2-56f3fe-kserve-self-signed-certs\"" Oct 30 21:58:00.947127 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:00.947102 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc"] Oct 30 21:58:01.023723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.023687 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:58:01.023986 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.023949 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:58:01.026121 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026085 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lhrg\" (UniqueName: \"kubernetes.io/projected/a377639d-cd1a-4182-8067-fb2071641be1-kube-api-access-5lhrg\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.026251 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026186 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-dshm\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.026251 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026216 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-tmp-dir\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.026343 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026328 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a377639d-cd1a-4182-8067-fb2071641be1-tls-certs\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.026400 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026356 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-home\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.026472 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026444 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-model-cache\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.026698 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026676 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-tmp-dir\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.026794 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026770 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-home\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.026832 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.026805 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-model-cache\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.028878 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.028848 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-dshm\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.029784 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.029761 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a377639d-cd1a-4182-8067-fb2071641be1-tls-certs\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.089731 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.089674 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lhrg\" (UniqueName: \"kubernetes.io/projected/a377639d-cd1a-4182-8067-fb2071641be1-kube-api-access-5lhrg\") pod \"distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.185490 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.178650 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:01.569049 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:01.569004 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc"] Oct 30 21:58:01.572558 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:58:01.572530 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda377639d_cd1a_4182_8067_fb2071641be1.slice/crio-42293e25712820b3f27668df46f640464d1cfd41e651ca829280f45e3be044fe WatchSource:0}: Error finding container 42293e25712820b3f27668df46f640464d1cfd41e651ca829280f45e3be044fe: Status 404 returned error can't find the container with id 42293e25712820b3f27668df46f640464d1cfd41e651ca829280f45e3be044fe Oct 30 21:58:02.023229 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:02.023196 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 30 21:58:02.023463 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:02.023245 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 30 21:58:02.026195 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:02.026154 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" event={"ID":"a377639d-cd1a-4182-8067-fb2071641be1","Type":"ContainerStarted","Data":"3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9"} Oct 30 21:58:02.026643 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:02.026206 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" event={"ID":"a377639d-cd1a-4182-8067-fb2071641be1","Type":"ContainerStarted","Data":"42293e25712820b3f27668df46f640464d1cfd41e651ca829280f45e3be044fe"} Oct 30 21:58:02.140376 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:02.140300 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podStartSLOduration=2.140283483 podStartE2EDuration="2.140283483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:02.136864976 +0000 UTC m=+1190.728150462" watchObservedRunningTime="2026-10-30 21:58:02.140283483 +0000 UTC m=+1190.731568969" Oct 30 21:58:02.180029 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:02.179991 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:03.027940 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:03.027909 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:58:03.028461 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:03.028340 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:58:03.028972 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:03.028951 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 21:58:11.179734 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:11.179696 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:11.192756 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:11.192724 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:12.166524 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:12.166484 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:58:19.804245 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:58:19.804207 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:58:28.437640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:28.437576 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:58:28.438223 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:28.437658 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:58:43.869974 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:43.869937 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd"] Oct 30 21:58:43.874261 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:43.874238 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:43.971739 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:43.971697 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd"] Oct 30 21:58:44.051655 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.051616 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.051655 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.051653 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.051871 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.051677 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.051871 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.051751 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.051938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.051872 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.051938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.051924 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.052062 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.052044 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zth7k\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-kube-api-access-zth7k\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.052098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.052076 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.052098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.052095 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.152875 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.152779 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zth7k\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-kube-api-access-zth7k\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.152875 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.152832 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.152875 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.152859 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153172 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.152890 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153172 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.152909 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153172 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.152932 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153172 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.152953 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153172 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.153021 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153172 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.153087 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153540 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.153339 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153540 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.153467 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153622 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.153533 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.153622 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.153580 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.154049 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.154026 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.155482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.155453 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.155670 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.155647 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.210752 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.210723 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.210905 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.210725 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zth7k\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-kube-api-access-zth7k\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.225353 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.225319 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4"] Oct 30 21:58:44.232026 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.231998 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.289187 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.289154 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4"] Oct 30 21:58:44.289966 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.289930 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"openshift-service-ca.crt\"" Oct 30 21:58:44.290129 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.289991 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"default-dockercfg-7bt4g\"" Oct 30 21:58:44.290129 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.290034 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"distinct-w1-3c4802-kserve-self-signed-certs\"" Oct 30 21:58:44.290129 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.290042 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"kube-root-ca.crt\"" Oct 30 21:58:44.356318 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.356281 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-model-cache\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.356532 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.356399 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-home\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.356532 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.356463 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8dt\" (UniqueName: \"kubernetes.io/projected/41b1cccb-9e1f-4cae-8142-3072eed003ea-kube-api-access-js8dt\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.356618 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.356573 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-dshm\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.356618 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.356601 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-tmp-dir\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.356681 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.356619 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b1cccb-9e1f-4cae-8142-3072eed003ea-tls-certs\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.457712 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.457620 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-model-cache\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.457888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.457763 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-home\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.457888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.457786 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8dt\" (UniqueName: \"kubernetes.io/projected/41b1cccb-9e1f-4cae-8142-3072eed003ea-kube-api-access-js8dt\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.457888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.457834 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-dshm\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.457888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.457853 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-tmp-dir\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.457888 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.457872 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b1cccb-9e1f-4cae-8142-3072eed003ea-tls-certs\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.458160 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.458088 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-model-cache\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.458213 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.458167 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-home\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.458290 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.458265 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-tmp-dir\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.460932 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.460907 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-dshm\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.461247 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.461229 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b1cccb-9e1f-4cae-8142-3072eed003ea-tls-certs\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.487424 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.487393 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:44.516749 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.516714 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8dt\" (UniqueName: \"kubernetes.io/projected/41b1cccb-9e1f-4cae-8142-3072eed003ea-kube-api-access-js8dt\") pod \"distinct-w1-3c4802-kserve-dc9d5b48-nq2k4\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.545849 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.545812 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:44.933308 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.933268 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd"] Oct 30 21:58:44.935637 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:58:44.935605 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34e3c031_6eef_46bc_a3c8_0ac1ae5ff92a.slice/crio-235c5dfacff1f70f76f15e521acbbc522fd173f715fedfd2efbffc8a1144233e WatchSource:0}: Error finding container 235c5dfacff1f70f76f15e521acbbc522fd173f715fedfd2efbffc8a1144233e: Status 404 returned error can't find the container with id 235c5dfacff1f70f76f15e521acbbc522fd173f715fedfd2efbffc8a1144233e Oct 30 21:58:44.938032 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.938000 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:44.938096 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.938067 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:44.938133 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.938100 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:58:44.947306 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:44.947283 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4"] Oct 30 21:58:44.981541 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:58:44.981512 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41b1cccb_9e1f_4cae_8142_3072eed003ea.slice/crio-4300c92c4f54684cdb15ffad2f06eeeed586e70dfa451afeafba0c00c92df1d1 WatchSource:0}: Error finding container 4300c92c4f54684cdb15ffad2f06eeeed586e70dfa451afeafba0c00c92df1d1: Status 404 returned error can't find the container with id 4300c92c4f54684cdb15ffad2f06eeeed586e70dfa451afeafba0c00c92df1d1 Oct 30 21:58:45.421483 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.421443 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" event={"ID":"41b1cccb-9e1f-4cae-8142-3072eed003ea","Type":"ContainerStarted","Data":"1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f"} Oct 30 21:58:45.421687 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.421488 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" event={"ID":"41b1cccb-9e1f-4cae-8142-3072eed003ea","Type":"ContainerStarted","Data":"4300c92c4f54684cdb15ffad2f06eeeed586e70dfa451afeafba0c00c92df1d1"} Oct 30 21:58:45.423042 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.423014 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" event={"ID":"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a","Type":"ContainerStarted","Data":"007a5f147c9273c701fb432205b59e2be13340e3b3c06e71f4dd579c1a6df976"} Oct 30 21:58:45.423168 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.423062 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" event={"ID":"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a","Type":"ContainerStarted","Data":"235c5dfacff1f70f76f15e521acbbc522fd173f715fedfd2efbffc8a1144233e"} Oct 30 21:58:45.488587 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.488553 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:45.490402 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.490374 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:58:45.490552 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.490447 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:58:45.532659 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.532373 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podStartSLOduration=1.532359316 podStartE2EDuration="1.532359316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:45.530707029 +0000 UTC m=+1234.121992515" watchObservedRunningTime="2026-10-30 21:58:45.532359316 +0000 UTC m=+1234.123644802" Oct 30 21:58:45.546821 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.546546 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:45.649313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:45.648278 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podStartSLOduration=2.648253687 podStartE2EDuration="2.648253687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:45.645729762 +0000 UTC m=+1234.237015249" watchObservedRunningTime="2026-10-30 21:58:45.648253687 +0000 UTC m=+1234.239539175" Oct 30 21:58:46.487913 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:46.487877 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:58:46.488363 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:46.487934 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:58:47.487903 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:47.487858 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:58:47.488096 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:47.487929 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:58:48.487915 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:48.487872 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:58:48.488117 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:48.487945 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:58:49.489091 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:49.488570 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:58:49.489091 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:49.488631 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:58:49.812329 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:58:49.812245 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:58:50.487925 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:50.487890 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:58:50.488085 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:50.487944 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:58:51.488016 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:51.487984 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 21:58:51.488388 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:51.488030 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 21:58:52.492287 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:52.492257 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:52.492712 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:52.492607 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:52.493466 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:52.493448 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 21:58:54.547090 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:54.547050 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:54.560521 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:54.560494 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:58:55.487374 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:58:55.487341 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:59:19.804336 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:59:19.804292 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:59:24.285533 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.285448 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr"] Oct 30 21:59:24.295638 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.295609 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.384974 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.384938 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr"] Oct 30 21:59:24.413806 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.413771 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.413969 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.413813 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.413969 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.413943 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.414051 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.413992 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.414095 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.414044 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.414176 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.414159 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.414266 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.414252 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.414307 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.414294 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spfr7\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-kube-api-access-spfr7\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.414347 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.414330 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.515112 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.515077 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.515313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.515127 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.515313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.515155 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.515313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.515192 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.515313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.515270 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.515618 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.515591 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.515696 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.515663 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.515758 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.515710 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.516098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.516074 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.516204 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.516132 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spfr7\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-kube-api-access-spfr7\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.516204 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.516179 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.516363 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.516343 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.516445 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.516375 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.516687 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.516648 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.518253 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.518225 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.518372 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.518350 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.574089 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.574011 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.574259 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.574236 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spfr7\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-kube-api-access-spfr7\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:24.608907 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:24.608880 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:25.031248 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.031221 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr"] Oct 30 21:59:25.033504 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:59:25.033471 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fe5a42_bf93_49b0_bb7e_c621ed3b867f.slice/crio-5b20094167df12ef952b74e1c50e9419d2b34e021ca7cf3639b59922e79ab87c WatchSource:0}: Error finding container 5b20094167df12ef952b74e1c50e9419d2b34e021ca7cf3639b59922e79ab87c: Status 404 returned error can't find the container with id 5b20094167df12ef952b74e1c50e9419d2b34e021ca7cf3639b59922e79ab87c Oct 30 21:59:25.035958 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.035917 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:25.036054 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.035995 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:25.036054 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.036028 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:25.423733 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.423654 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr"] Oct 30 21:59:25.427649 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.427624 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.469640 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.469610 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj"] Oct 30 21:59:25.473513 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.473485 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.488073 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.488043 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr"] Oct 30 21:59:25.531151 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531115 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.531355 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531215 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.531355 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531241 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e91287-9971-475c-a10f-c95f3f346b59-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.531355 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531271 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.531592 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531442 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7z9p\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-kube-api-access-r7z9p\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.531592 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531495 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.531592 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531519 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.531731 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531630 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.531731 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531705 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.531824 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531737 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e91287-9971-475c-a10f-c95f3f346b59-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.531824 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531767 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.531824 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531795 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.531938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531824 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.531938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531851 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfk5c\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-kube-api-access-kfk5c\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.531938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531901 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.531938 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531933 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.532081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.531963 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.532081 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.532005 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.556964 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.556931 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj"] Oct 30 21:59:25.633325 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633285 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.633542 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633362 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.633542 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633389 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e91287-9971-475c-a10f-c95f3f346b59-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.633542 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633431 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.633542 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633529 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7z9p\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-kube-api-access-r7z9p\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.633777 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633556 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.633834 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633784 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.633893 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633842 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.633951 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633890 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634004 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.633984 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.634058 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634018 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e91287-9971-475c-a10f-c95f3f346b59-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634056 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.634114 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634083 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.634278 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634120 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634278 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634145 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfk5c\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-kube-api-access-kfk5c\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634278 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634201 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634278 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634088 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.634278 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634230 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634278 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634260 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.634623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634298 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634349 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.634623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634514 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.634623 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634232 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e91287-9971-475c-a10f-c95f3f346b59-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634839 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634644 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.634839 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634778 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634839 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634804 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.634980 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.634882 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.635140 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.635113 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.636796 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.636761 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.636908 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.636826 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.636975 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.636947 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.637173 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.637156 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e91287-9971-475c-a10f-c95f3f346b59-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.698569 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.698472 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" event={"ID":"19fe5a42-bf93-49b0-bb7e-c621ed3b867f","Type":"ContainerStarted","Data":"2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5"} Oct 30 21:59:25.698569 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.698535 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" event={"ID":"19fe5a42-bf93-49b0-bb7e-c621ed3b867f","Type":"ContainerStarted","Data":"5b20094167df12ef952b74e1c50e9419d2b34e021ca7cf3639b59922e79ab87c"} Oct 30 21:59:25.702921 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.702891 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.703063 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.702891 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.703063 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.702938 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfk5c\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-kube-api-access-kfk5c\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.703063 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.702900 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7z9p\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-kube-api-access-r7z9p\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.743939 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.743911 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:25.789860 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.789822 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:25.816585 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:25.816530 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podStartSLOduration=1.816509812 podStartE2EDuration="1.816509812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:25.815385109 +0000 UTC m=+1274.406670596" watchObservedRunningTime="2026-10-30 21:59:25.816509812 +0000 UTC m=+1274.407795360" Oct 30 21:59:26.199164 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.199124 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr"] Oct 30 21:59:26.202565 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:59:26.202535 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e91287_9971_475c_a10f_c95f3f346b59.slice/crio-9456f35653e64344823023002185ecf66b4cc8dc2e87eff4d0be63b3c3b88a0e WatchSource:0}: Error finding container 9456f35653e64344823023002185ecf66b4cc8dc2e87eff4d0be63b3c3b88a0e: Status 404 returned error can't find the container with id 9456f35653e64344823023002185ecf66b4cc8dc2e87eff4d0be63b3c3b88a0e Oct 30 21:59:26.209138 ip-10-0-133-5 kubenswrapper[5139]: W1030 21:59:26.209109 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfe38e7c_911f_432d_a9b6_90bc9b31de5c.slice/crio-7c5653c1ee7a5ce36486766becb9fb327abade2af1371d53fa329bc8f899eea2 WatchSource:0}: Error finding container 7c5653c1ee7a5ce36486766becb9fb327abade2af1371d53fa329bc8f899eea2: Status 404 returned error can't find the container with id 7c5653c1ee7a5ce36486766becb9fb327abade2af1371d53fa329bc8f899eea2 Oct 30 21:59:26.209689 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.209655 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.209784 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.209740 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.209784 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.209771 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.210314 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.210217 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj"] Oct 30 21:59:26.211761 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.211717 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.211832 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.211797 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.211873 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.211853 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:26.610136 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.609934 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:26.612389 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.612354 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:26.612568 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.612448 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:26.707029 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.706990 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" event={"ID":"cfe38e7c-911f-432d-a9b6-90bc9b31de5c","Type":"ContainerStarted","Data":"f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d"} Oct 30 21:59:26.707241 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.707069 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" event={"ID":"cfe38e7c-911f-432d-a9b6-90bc9b31de5c","Type":"ContainerStarted","Data":"7c5653c1ee7a5ce36486766becb9fb327abade2af1371d53fa329bc8f899eea2"} Oct 30 21:59:26.708874 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.708842 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" event={"ID":"b2e91287-9971-475c-a10f-c95f3f346b59","Type":"ContainerStarted","Data":"65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7"} Oct 30 21:59:26.709018 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.708893 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" event={"ID":"b2e91287-9971-475c-a10f-c95f3f346b59","Type":"ContainerStarted","Data":"9456f35653e64344823023002185ecf66b4cc8dc2e87eff4d0be63b3c3b88a0e"} Oct 30 21:59:26.743991 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.743961 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:26.795438 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.793715 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:26.822779 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.822697 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podStartSLOduration=1.8226808220000001 podStartE2EDuration="1.822680822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:26.821526459 +0000 UTC m=+1275.412811945" watchObservedRunningTime="2026-10-30 21:59:26.822680822 +0000 UTC m=+1275.413966309" Oct 30 21:59:26.954503 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:26.954374 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podStartSLOduration=1.954360012 podStartE2EDuration="1.954360012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:26.951890132 +0000 UTC m=+1275.543175662" watchObservedRunningTime="2026-10-30 21:59:26.954360012 +0000 UTC m=+1275.545645498" Oct 30 21:59:27.270439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.265592 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw"] Oct 30 21:59:27.290440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.278905 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402131 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402203 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402297 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402327 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402351 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402878 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402490 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2k5t\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-kube-api-access-c2k5t\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402878 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402546 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402878 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402587 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.402878 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.402633 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.431440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.423175 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw"] Oct 30 21:59:27.508450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508110 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2k5t\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-kube-api-access-c2k5t\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.508450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508198 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.508450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508240 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.508450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508287 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.508450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508356 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.508450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508398 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.508953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508510 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.508953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508545 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.508953 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.508571 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.550500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.509125 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.550500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.510001 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.550500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.510267 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.550500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.533922 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.584441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.583630 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.665450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.616666 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.665450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.619899 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:27.665450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.619958 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:27.665450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.631915 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.665450 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.656370 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2k5t\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-kube-api-access-c2k5t\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.704440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.677988 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:27.747917 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.747880 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:27.748089 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.747957 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:27.801500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.800560 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:27.801500 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.800644 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:27.910720 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:27.910665 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:28.521440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.515119 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw"] Oct 30 21:59:28.547989 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.547501 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:28.547989 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.547595 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:28.547989 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.547640 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:28.609338 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.609261 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:28.609619 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.609544 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:28.744796 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.744743 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:28.745463 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.744824 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:28.754854 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.754820 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" event={"ID":"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c","Type":"ContainerStarted","Data":"9631f85b14457712b1ce36c2ca69dc4a4e8726493143013f5397104363ff3025"} Oct 30 21:59:28.793529 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.791807 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:28.793529 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:28.791875 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:29.299390 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.299348 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs"] Oct 30 21:59:29.306773 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.306728 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458302 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458362 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nspwn\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-kube-api-access-nspwn\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458448 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458525 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458610 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458678 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458780 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458823 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.459436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.458849 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568318 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568402 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568531 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568578 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568605 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568690 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568725 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nspwn\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-kube-api-access-nspwn\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568805 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.568881 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.569703 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.569989 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.570243 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.570567 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.571482 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.571231 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.801407 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.801369 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:29.809880 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.809838 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:29.810504 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.802946 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:29.810678 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.810655 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:29.810894 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.803027 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:29.811045 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.811023 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:29.963559 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.963467 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.973462 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.964059 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.973462 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.964630 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.973462 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.968069 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nspwn\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-kube-api-access-nspwn\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:29.975378 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.975352 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:29.975628 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.975587 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:29.988473 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.981482 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" event={"ID":"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c","Type":"ContainerStarted","Data":"01c80f9382d8e8222d0b6ec48b4cb1c0890a895557b7e8c122e9d29f0df2671e"} Oct 30 21:59:29.988473 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.981565 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:29.988473 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.981608 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs"] Oct 30 21:59:29.997454 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:29.996172 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:30.048438 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.045648 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podStartSLOduration=3.045628423 podStartE2EDuration="3.045628423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:30.030933424 +0000 UTC m=+1278.622218914" watchObservedRunningTime="2026-10-30 21:59:30.045628423 +0000 UTC m=+1278.636913911" Oct 30 21:59:30.096441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.076519 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 21:59:30.177437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.176480 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:59:30.177437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.176543 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:30.271435 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.271208 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:30.390294 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.389500 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:59:30.408862 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.403530 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:30.485817 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.485769 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:30.611971 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.611803 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:30.611971 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.611864 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.999245 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:30.999652 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.000743 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.000791 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.002060 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.002096 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.003658 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.003690 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:31.012439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.003749 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:31.247439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.244850 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:31.247439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.245170 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:59:31.247439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.246888 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:59:31.247439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.246959 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:59:31.341178 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.341089 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs"] Oct 30 21:59:31.421572 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.421520 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:31.451446 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.423827 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:59:31.482456 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.426302 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:31.507464 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.507430 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:59:31.608498 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.576539 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:31.654443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.654274 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.654443 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.654333 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:31.734449 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.730792 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:31.734449 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.730888 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:31.734449 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.730935 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 21:59:31.803313 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.803275 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.803612 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.803580 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:31.987496 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.977487 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:31.987496 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:31.977568 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:32.008552 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.008495 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:32.008837 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.008796 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:32.083790 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.083734 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 21:59:32.147479 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.147434 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:32.219380 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.219334 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:32.251041 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.251005 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:59:32.251344 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.251327 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:59:32.255607 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.255587 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:59:32.255793 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.255783 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:59:32.282442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.260244 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded" Oct 30 21:59:32.284391 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.266503 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:32.284909 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.266539 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:59:32.285792 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.282758 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:32.427466 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.427399 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:59:32.533525 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.531868 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:59:32.589302 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.589256 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:32.590250 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.590208 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" event={"ID":"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed","Type":"ContainerStarted","Data":"3638676d96c5a37ade9110dd330cdcf0ac040581f5e4757502feb85e0494107a"} Oct 30 21:59:32.766717 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.766680 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:32.766958 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.766930 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:32.804491 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.779064 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:32.805240 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.805204 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:32.805557 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.790324 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:32.805557 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:32.805499 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:33.078826 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.078711 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:33.079123 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.079065 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:33.087228 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.087191 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 21:59:33.155552 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.155505 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:33.229459 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.221507 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:33.291551 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.291498 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:33.304446 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.292191 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:33.331492 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.306189 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:33.411468 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.393332 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:59:33.514852 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.514810 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:33.845499 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.823636 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:59:34.000499 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.991886 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:34.000499 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:33.991943 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:34.020446 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.001927 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:34.020446 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.002005 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:34.120624 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.098173 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 21:59:34.120624 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.111045 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:34.120624 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.113967 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:34.120624 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.114016 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:34.204403 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.180189 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:34.290632 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.186662 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:34.291688 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.291650 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:34.291903 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.224894 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:34.295437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.224808 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/health\": context deadline exceeded" Oct 30 21:59:34.299531 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.297541 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:34.332887 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.310475 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:34.377442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.347021 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" event={"ID":"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed","Type":"ContainerStarted","Data":"346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff"} Oct 30 21:59:34.395780 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.395724 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:34.434218 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.434140 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podStartSLOduration=5.43412026 podStartE2EDuration="5.43412026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:34.432023257 +0000 UTC m=+1283.023308744" watchObservedRunningTime="2026-10-30 21:59:34.43412026 +0000 UTC m=+1283.025405747" Oct 30 21:59:34.517665 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.517620 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:34.654494 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.619773 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:34.654494 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.619838 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:34.773655 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.770562 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:34.773655 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.770635 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:34.790190 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.790153 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:34.790521 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.790450 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:34.835399 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.834545 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:34.930528 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.930404 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:34.930528 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:34.930492 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:35.004408 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.004378 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:35.004812 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.004774 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:59:35.004981 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.004958 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:59:35.054316 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.054288 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:59:35.092288 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.092244 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 21:59:35.092672 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.092547 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:35.098993 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.098958 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:35.230759 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.230660 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:35.231209 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.230756 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 21:59:35.301563 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.301512 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:59:35.371440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.341538 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:35.617527 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.376533 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:35.618786 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.401149 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:35.625116 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.519802 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:35.634507 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.553523 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:35.834808 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.834726 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:59:35.889336 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.889237 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 21:59:36.136608 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.136545 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 21:59:36.136914 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.101594 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:36.137980 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.913202 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:36.138186 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.138148 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:36.138321 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.913255 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:36.138456 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.138436 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:36.138596 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:35.938273 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:36.138731 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.138709 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:36.142589 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.024838 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:59:36.142766 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.142736 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:59:36.231568 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.231515 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:36.327368 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.327326 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:36.330983 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.328601 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:59:36.625524 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.625479 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:59:36.638586 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.628210 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:36.638586 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.628876 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:36.638586 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.629099 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 21:59:36.755339 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.755290 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:36.762564 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.762499 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:36.794100 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.793970 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:36.794100 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.794030 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:36.854453 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.848528 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:36.921497 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.920789 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:36.921497 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.920852 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:37.000441 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:36.999994 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:37.000955 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.000927 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:59:37.010444 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.001156 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:59:37.010689 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.010112 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:37.081574 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.081544 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 21:59:37.127451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.125882 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:59:37.147437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.143695 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded" Oct 30 21:59:37.248633 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.248590 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:37.277440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.270707 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:59:37.333845 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.333799 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:37.402437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.334077 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:37.402437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.385516 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:59:37.603526 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.582756 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": context deadline exceeded" Oct 30 21:59:37.636149 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.636092 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:59:37.640663 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.640619 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:37.714659 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.695866 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded" Oct 30 21:59:37.860095 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.843601 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 21:59:37.860345 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.860309 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 21:59:37.860522 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.855706 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 21:59:37.860663 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.860643 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 21:59:37.933594 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.926911 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:37.933594 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:37.926967 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:38.027440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.001557 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:59:38.029557 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.029484 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:59:38.157596 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.157499 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:38.254570 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.254506 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 21:59:38.387130 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.387087 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:38.387699 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.387670 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:59:38.527437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.526917 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 21:59:38.833491 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.832883 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:38.833491 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.833199 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 21:59:38.852064 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.852015 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:38.861161 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.861122 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:38.917375 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.917338 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:38.917695 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:38.917666 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:39.017220 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.017188 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:59:39.017543 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.017512 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:59:39.178941 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.178828 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:39.242661 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.242532 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:39.257383 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.256521 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:39.394626 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.394578 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:39.401919 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.401876 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 21:59:39.537644 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.537599 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:39.538459 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.538438 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:59:39.645440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.636435 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 21:59:39.652098 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.652073 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:59:39.702008 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.701975 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:59:39.754532 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.754495 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:59:39.754771 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.754746 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:39.846501 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.845297 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:59:39.846980 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.846696 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:39.911847 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.911810 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 21:59:39.912097 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.912073 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 21:59:39.942714 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.942687 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:59:39.959451 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:39.959409 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:59:40.006892 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.006853 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:59:40.007146 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.007115 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:59:40.028468 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.028295 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:59:40.046006 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.039587 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:59:40.147842 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.147758 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:59:40.762460 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.762155 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:40.832438 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.801508 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:59:40.832438 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.801577 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:40.832438 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.802133 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:40.941439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.941151 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:59:40.941439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:40.941208 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:41.017498 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.016545 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:59:41.017498 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.016611 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:59:41.286499 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.286172 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:41.342862 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.342826 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:41.343663 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.343271 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:41.344459 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.344429 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 21:59:41.452786 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.452746 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:41.854305 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.854273 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 21:59:41.854710 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.854385 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 21:59:41.998336 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.998201 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 21:59:41.998336 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:41.998257 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 21:59:43.078760 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:43.078726 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:43.079549 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:43.079318 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:43.080605 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:43.080584 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 21:59:49.805205 ip-10-0-133-5 kubenswrapper[5139]: E1030 21:59:49.805162 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 21:59:57.560442 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:57.559922 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:57.593439 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:57.592743 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:57.641437 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:57.641229 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:57.828499 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:57.827576 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.054579 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.054518 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.099715 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.099615 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 21:59:58.291436 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.288569 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.292754 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.292713 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.403892 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.403784 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.406710 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.406677 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.444026 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.437531 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:59:58.444026 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.437624 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.595539 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.595491 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.596299 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.596267 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:59:58.690656 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.690556 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:58.691284 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.691258 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:59:58.925043 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.924999 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 21:59:58.950873 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:58.925930 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 21:59:59.056578 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.056534 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:59.073770 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.073734 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:59:59.124440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.102506 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:59.124440 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.102803 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:59:59.146857 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.146816 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 21:59:59.147129 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.147099 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:59.221723 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.221616 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:59.332709 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.332656 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:59.336099 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.334737 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:59.336373 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.336350 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:59:59.342034 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.342011 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:59:59.492521 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.489541 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 21:59:59.501074 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.499275 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": context deadline exceeded" Oct 30 21:59:59.501074 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.500374 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:59:59.614751 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.608493 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:59.692345 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.691518 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 21:59:59.738491 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.735844 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" Oct 30 21:59:59.738491 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.735909 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 21:59:59.743909 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.743834 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 21:59:59.774631 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.774603 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 21:59:59.855771 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.854139 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 21:59:59.856508 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.856485 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 21:59:59.857548 ip-10-0-133-5 kubenswrapper[5139]: I1030 21:59:59.857521 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 22:00:19.805519 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:00:19.805468 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:00:24.789315 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.789279 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw"] Oct 30 22:00:24.794587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.794556 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.867542 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.867503 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.867730 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.867559 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d25c30-87c6-4eff-857c-87973d3bbfab-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.867730 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.867579 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.867847 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.867778 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.867958 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.867928 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.868056 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.867987 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.868056 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.868040 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.868177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.868061 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.868177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.868098 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjdkk\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-kube-api-access-bjdkk\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.968956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.968922 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969046 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969085 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969149 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969180 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969223 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjdkk\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-kube-api-access-bjdkk\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969272 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969370 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969466 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d25c30-87c6-4eff-857c-87973d3bbfab-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969522 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969526 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969607 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.969881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.969641 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.970150 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.970127 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d25c30-87c6-4eff-857c-87973d3bbfab-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.971356 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.971334 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:24.971612 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:24.971593 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:25.002555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:25.002514 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw"] Oct 30 22:00:25.029644 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:25.029609 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjdkk\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-kube-api-access-bjdkk\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:25.030297 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:25.030273 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:25.108687 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:25.108585 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:25.531956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:25.531925 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw"] Oct 30 22:00:25.533407 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:00:25.533370 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21d25c30_87c6_4eff_857c_87973d3bbfab.slice/crio-b3fe8ed6805235c55caffd928dbd59cc17dc1910f2720f65c66f2540e8738d01 WatchSource:0}: Error finding container b3fe8ed6805235c55caffd928dbd59cc17dc1910f2720f65c66f2540e8738d01: Status 404 returned error can't find the container with id b3fe8ed6805235c55caffd928dbd59cc17dc1910f2720f65c66f2540e8738d01 Oct 30 22:00:25.535962 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:25.535927 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:25.536062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:25.535991 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:25.536062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:25.536021 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:00:26.296287 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:26.296256 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" event={"ID":"21d25c30-87c6-4eff-857c-87973d3bbfab","Type":"ContainerStarted","Data":"665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d"} Oct 30 22:00:26.296835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:26.296304 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" event={"ID":"21d25c30-87c6-4eff-857c-87973d3bbfab","Type":"ContainerStarted","Data":"b3fe8ed6805235c55caffd928dbd59cc17dc1910f2720f65c66f2540e8738d01"} Oct 30 22:00:26.460387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:26.460336 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podStartSLOduration=2.46032193 podStartE2EDuration="2.46032193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:26.45876557 +0000 UTC m=+1335.050051057" watchObservedRunningTime="2026-10-30 22:00:26.46032193 +0000 UTC m=+1335.051607416" Oct 30 22:00:27.108992 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:27.108952 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:27.110788 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:27.110760 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:00:27.110915 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:27.110814 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:00:28.110021 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:28.109982 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:00:28.110460 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:28.110053 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:00:29.109078 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:29.109041 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:00:29.109299 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:29.109116 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:00:30.109864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:30.109817 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:00:30.110275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:30.109907 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:00:31.123442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:31.113001 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:00:31.123442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:31.113065 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:00:32.110853 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:32.110807 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:00:32.111043 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:32.110889 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:00:34.152569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:34.152526 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:00:34.153012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:34.152576 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:00:34.607613 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:34.607579 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:34.608097 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:34.608055 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:34.609309 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:34.609269 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:00:49.806304 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:00:49.806266 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:00:51.004162 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.004067 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w"] Oct 30 22:00:51.008212 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.008187 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.037534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.037505 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8feac-77b9-47a9-bf58-8c815ca853a4-openshift-service-ca\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.037661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.037577 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-metrics-certs\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.037726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.037705 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.037878 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.037859 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwqk\" (UniqueName: \"kubernetes.io/projected/8bb8feac-77b9-47a9-bf58-8c815ca853a4-kube-api-access-7xwqk\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.057791 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.057768 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-4c260505-serving-cert\"" Oct 30 22:00:51.107141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.107095 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w"] Oct 30 22:00:51.139004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.138960 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwqk\" (UniqueName: \"kubernetes.io/projected/8bb8feac-77b9-47a9-bf58-8c815ca853a4-kube-api-access-7xwqk\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.139175 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.139026 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8feac-77b9-47a9-bf58-8c815ca853a4-openshift-service-ca\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.139175 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.139100 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-metrics-certs\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.139319 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.139191 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.139371 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:00:51.139336 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-serving-cert: secret "maas-api-e2e-mt-4c260505-serving-cert" not found Oct 30 22:00:51.139470 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:00:51.139456 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls podName:8bb8feac-77b9-47a9-bf58-8c815ca853a4 nodeName:}" failed. No retries permitted until 2026-10-30 22:00:51.639402312 +0000 UTC m=+1360.230687795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls") pod "maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" (UID: "8bb8feac-77b9-47a9-bf58-8c815ca853a4") : secret "maas-api-e2e-mt-4c260505-serving-cert" not found Oct 30 22:00:51.139993 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.139966 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8feac-77b9-47a9-bf58-8c815ca853a4-openshift-service-ca\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.141869 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.141841 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-metrics-certs\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.197901 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.197870 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwqk\" (UniqueName: \"kubernetes.io/projected/8bb8feac-77b9-47a9-bf58-8c815ca853a4-kube-api-access-7xwqk\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.644972 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.644936 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.647541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.647517 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls\") pod \"maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:51.921612 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:51.921525 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:52.312635 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:52.312604 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w"] Oct 30 22:00:52.314386 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:00:52.314342 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb8feac_77b9_47a9_bf58_8c815ca853a4.slice/crio-e947f93474c7f5dfd6001dccb66f81182edbeca8e5b4b4d56f7982b37e01ec26 WatchSource:0}: Error finding container e947f93474c7f5dfd6001dccb66f81182edbeca8e5b4b4d56f7982b37e01ec26: Status 404 returned error can't find the container with id e947f93474c7f5dfd6001dccb66f81182edbeca8e5b4b4d56f7982b37e01ec26 Oct 30 22:00:52.315858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:52.315833 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:00:52.491496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:52.491459 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" event={"ID":"8bb8feac-77b9-47a9-bf58-8c815ca853a4","Type":"ContainerStarted","Data":"e947f93474c7f5dfd6001dccb66f81182edbeca8e5b4b4d56f7982b37e01ec26"} Oct 30 22:00:53.500105 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:53.500066 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" event={"ID":"8bb8feac-77b9-47a9-bf58-8c815ca853a4","Type":"ContainerStarted","Data":"2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3"} Oct 30 22:00:53.500617 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:53.500187 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:00:53.696294 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:53.696190 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" podStartSLOduration=2.70573109 podStartE2EDuration="3.696174228s" totalImagesPullingTime="990.443138ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:53.693131438 +0000 UTC m=+1362.284416933" watchObservedRunningTime="2026-10-30 22:00:53.696174228 +0000 UTC m=+1362.287459715" Oct 30 22:00:59.511798 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:00:59.511767 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:01:13.122152 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.122116 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd"] Oct 30 22:01:13.126322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.126299 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.228827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.228788 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plf48\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-kube-api-access-plf48\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.229014 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.228930 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.229014 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.228968 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.229108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.229016 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.229158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.229106 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.229277 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.229255 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.229350 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.229315 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.229408 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.229384 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94b7c7c-9e17-4087-b287-ffe225700146-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.229485 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.229470 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94b7c7c-9e17-4087-b287-ffe225700146-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.330968 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.330924 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plf48\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-kube-api-access-plf48\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331052 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331079 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331120 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331146 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331409 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331248 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331409 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331310 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331409 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331350 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94b7c7c-9e17-4087-b287-ffe225700146-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331409 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331384 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94b7c7c-9e17-4087-b287-ffe225700146-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331668 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331577 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331783 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331756 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331853 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331799 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.331985 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.331961 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.332270 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.332226 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94b7c7c-9e17-4087-b287-ffe225700146-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.333243 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.333221 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd"] Oct 30 22:01:13.335155 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.334578 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94b7c7c-9e17-4087-b287-ffe225700146-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.337189 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.336935 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.388913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.388842 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.388913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.388895 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plf48\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-kube-api-access-plf48\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.441270 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.441238 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:13.868037 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.867997 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd"] Oct 30 22:01:13.899711 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:01:13.899676 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94b7c7c_9e17_4087_b287_ffe225700146.slice/crio-c13277d7b9856f2034fea7bdbe9faee2c268bcb30c203c9f9cf897dca69b7810 WatchSource:0}: Error finding container c13277d7b9856f2034fea7bdbe9faee2c268bcb30c203c9f9cf897dca69b7810: Status 404 returned error can't find the container with id c13277d7b9856f2034fea7bdbe9faee2c268bcb30c203c9f9cf897dca69b7810 Oct 30 22:01:13.902191 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.902155 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:01:13.902271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.902228 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:01:13.902271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:13.902256 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:01:14.632504 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:14.632469 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" event={"ID":"f94b7c7c-9e17-4087-b287-ffe225700146","Type":"ContainerStarted","Data":"59c36f7cb7f027b2af63017c3f1cf970dafeb956288aa43141eec537deff2320"} Oct 30 22:01:14.633001 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:14.632536 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" event={"ID":"f94b7c7c-9e17-4087-b287-ffe225700146","Type":"ContainerStarted","Data":"c13277d7b9856f2034fea7bdbe9faee2c268bcb30c203c9f9cf897dca69b7810"} Oct 30 22:01:14.796124 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:14.796067 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" podStartSLOduration=1.796051196 podStartE2EDuration="1.796051196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:14.795100413 +0000 UTC m=+1383.386385925" watchObservedRunningTime="2026-10-30 22:01:14.796051196 +0000 UTC m=+1383.387336682" Oct 30 22:01:15.441788 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:15.441752 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:15.443534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:15.443509 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:01:15.443668 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:15.443556 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:01:16.442665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:16.442623 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:01:16.443172 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:16.442684 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:01:16.889899 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:16.889862 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6"] Oct 30 22:01:16.894277 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:16.894242 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:16.942856 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:16.942823 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6"] Oct 30 22:01:16.987442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:16.986771 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"distinct-2-w4-26743a-kserve-self-signed-certs\"" Oct 30 22:01:17.001426 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.001374 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tls-certs\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.001794 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.001778 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-home\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.001877 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.001855 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2jxl\" (UniqueName: \"kubernetes.io/projected/1dbb1410-3b3b-4a28-bb15-92efac37aa44-kube-api-access-z2jxl\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.002031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.002015 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tmp-dir\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.002067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.002052 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-dshm\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.002184 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.002162 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-model-cache\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.103389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.103356 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tls-certs\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.103593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.103433 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-home\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.103593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.103457 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2jxl\" (UniqueName: \"kubernetes.io/projected/1dbb1410-3b3b-4a28-bb15-92efac37aa44-kube-api-access-z2jxl\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.103593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.103543 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tmp-dir\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.103593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.103567 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-dshm\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.103778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.103628 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-model-cache\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.103967 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.103939 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-home\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.104086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.103966 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tmp-dir\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.104086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.104001 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-model-cache\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.105987 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.105965 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tls-certs\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.106113 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.105989 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-dshm\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.197369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.197278 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2jxl\" (UniqueName: \"kubernetes.io/projected/1dbb1410-3b3b-4a28-bb15-92efac37aa44-kube-api-access-z2jxl\") pod \"distinct-2-w4-26743a-kserve-76599f66b5-9kph6\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.207592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.207559 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:17.442008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.441975 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:01:17.442194 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.442028 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:01:17.593436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.593390 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6"] Oct 30 22:01:17.595458 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:01:17.595430 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dbb1410_3b3b_4a28_bb15_92efac37aa44.slice/crio-8b7a42080a1e33b3b6d63028ddb5cf0832d78871a3aaa0280cd6b404295d9e2e WatchSource:0}: Error finding container 8b7a42080a1e33b3b6d63028ddb5cf0832d78871a3aaa0280cd6b404295d9e2e: Status 404 returned error can't find the container with id 8b7a42080a1e33b3b6d63028ddb5cf0832d78871a3aaa0280cd6b404295d9e2e Oct 30 22:01:17.657910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:17.657876 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" event={"ID":"1dbb1410-3b3b-4a28-bb15-92efac37aa44","Type":"ContainerStarted","Data":"8b7a42080a1e33b3b6d63028ddb5cf0832d78871a3aaa0280cd6b404295d9e2e"} Oct 30 22:01:18.442362 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:18.442325 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:01:18.442643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:18.442377 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:01:18.694280 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:18.694184 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" event={"ID":"1dbb1410-3b3b-4a28-bb15-92efac37aa44","Type":"ContainerStarted","Data":"af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5"} Oct 30 22:01:18.809213 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:18.809161 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" podStartSLOduration=2.809148126 podStartE2EDuration="2.809148126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:18.808229689 +0000 UTC m=+1387.399515177" watchObservedRunningTime="2026-10-30 22:01:18.809148126 +0000 UTC m=+1387.400433612" Oct 30 22:01:19.207827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:19.207789 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:19.442605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:19.442565 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:01:19.442788 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:19.442627 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:01:19.814939 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:01:19.814834 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:01:20.442482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:20.442437 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 22:01:20.442667 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:20.442502 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 22:01:21.446115 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:21.446082 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:21.446540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:21.446373 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:21.446987 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:21.446968 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:01:27.208383 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:27.208325 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:27.221246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:27.221223 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:27.761158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:01:27.761122 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:01:49.804864 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:01:49.804824 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:02:10.232484 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.229702 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl"] Oct 30 22:02:10.243296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.243269 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:10.273316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.271921 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl"] Oct 30 22:02:10.313541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.313510 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-695f5c\"" Oct 30 22:02:10.314496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.313762 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-695f5c-dockercfg-7bnm8\"" Oct 30 22:02:10.342883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.342844 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01451298-22d3-48eb-a464-8ff1c7b32a31-plugins-config-volume\") pod \"payload-processing-e2e-api-a-695f5c-7698569c67-425hl\" (UID: \"01451298-22d3-48eb-a464-8ff1c7b32a31\") " pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:10.343091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.343068 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw8q6\" (UniqueName: \"kubernetes.io/projected/01451298-22d3-48eb-a464-8ff1c7b32a31-kube-api-access-cw8q6\") pod \"payload-processing-e2e-api-a-695f5c-7698569c67-425hl\" (UID: \"01451298-22d3-48eb-a464-8ff1c7b32a31\") " pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:10.444057 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.444020 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01451298-22d3-48eb-a464-8ff1c7b32a31-plugins-config-volume\") pod \"payload-processing-e2e-api-a-695f5c-7698569c67-425hl\" (UID: \"01451298-22d3-48eb-a464-8ff1c7b32a31\") " pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:10.444261 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.444206 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw8q6\" (UniqueName: \"kubernetes.io/projected/01451298-22d3-48eb-a464-8ff1c7b32a31-kube-api-access-cw8q6\") pod \"payload-processing-e2e-api-a-695f5c-7698569c67-425hl\" (UID: \"01451298-22d3-48eb-a464-8ff1c7b32a31\") " pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:10.444919 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.444896 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01451298-22d3-48eb-a464-8ff1c7b32a31-plugins-config-volume\") pod \"payload-processing-e2e-api-a-695f5c-7698569c67-425hl\" (UID: \"01451298-22d3-48eb-a464-8ff1c7b32a31\") " pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:10.502268 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.502241 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw8q6\" (UniqueName: \"kubernetes.io/projected/01451298-22d3-48eb-a464-8ff1c7b32a31-kube-api-access-cw8q6\") pod \"payload-processing-e2e-api-a-695f5c-7698569c67-425hl\" (UID: \"01451298-22d3-48eb-a464-8ff1c7b32a31\") " pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:10.577306 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.577274 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:10.947739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:10.947715 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl"] Oct 30 22:02:10.951049 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:02:10.951018 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01451298_22d3_48eb_a464_8ff1c7b32a31.slice/crio-359ba8ddd37fcbad8061b60e508a423aaaa5ddebcc1821e6b60276d68c03f34f WatchSource:0}: Error finding container 359ba8ddd37fcbad8061b60e508a423aaaa5ddebcc1821e6b60276d68c03f34f: Status 404 returned error can't find the container with id 359ba8ddd37fcbad8061b60e508a423aaaa5ddebcc1821e6b60276d68c03f34f Oct 30 22:02:11.060340 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:11.060304 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" event={"ID":"01451298-22d3-48eb-a464-8ff1c7b32a31","Type":"ContainerStarted","Data":"2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463"} Oct 30 22:02:11.060533 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:11.060387 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" event={"ID":"01451298-22d3-48eb-a464-8ff1c7b32a31","Type":"ContainerStarted","Data":"359ba8ddd37fcbad8061b60e508a423aaaa5ddebcc1821e6b60276d68c03f34f"} Oct 30 22:02:11.060533 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:11.060457 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:11.173008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:11.172953 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" podStartSLOduration=1.172937316 podStartE2EDuration="1.172937316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:11.169317937 +0000 UTC m=+1439.760603425" watchObservedRunningTime="2026-10-30 22:02:11.172937316 +0000 UTC m=+1439.764222802" Oct 30 22:02:13.624247 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.624211 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx"] Oct 30 22:02:13.628346 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.628328 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.711689 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.711655 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwlln\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-kube-api-access-fwlln\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.712013 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.711986 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.712478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.712282 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.712478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.712402 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.712478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.712460 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.712827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.712565 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.712827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.712597 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.712827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.712639 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.712827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.712676 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.730118 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.726090 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx"] Oct 30 22:02:13.813533 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813490 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.813778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813644 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.813778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813704 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.813778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813735 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.813953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813822 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwlln\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-kube-api-access-fwlln\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.813953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813859 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.814062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813937 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.814062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813976 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.814062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.813997 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.814216 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.814137 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.814216 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.814168 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.814321 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.814274 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.814753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.814728 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.814960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.814941 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.816253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.816227 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.816532 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.816512 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.871696 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.871660 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.871934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.871904 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwlln\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-kube-api-access-fwlln\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:13.948389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:13.948297 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:14.438322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:14.438293 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx"] Oct 30 22:02:14.440595 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:02:14.440565 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231eef6e_a1f5_44fa_a50f_ffcff69259dd.slice/crio-d01a5d18f692851d61f12fb70562f4e40b2185b087072645008f6de863013d8d WatchSource:0}: Error finding container d01a5d18f692851d61f12fb70562f4e40b2185b087072645008f6de863013d8d: Status 404 returned error can't find the container with id d01a5d18f692851d61f12fb70562f4e40b2185b087072645008f6de863013d8d Oct 30 22:02:14.443059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:14.443022 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:14.443132 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:14.443099 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:14.443177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:14.443130 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:15.093173 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:15.093133 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" event={"ID":"231eef6e-a1f5-44fa-a50f-ffcff69259dd","Type":"ContainerStarted","Data":"357f9438b4b943694b6bba9b8016000b3ff3f1a272a388359a0ae6a4f17e80ae"} Oct 30 22:02:15.093664 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:15.093188 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" event={"ID":"231eef6e-a1f5-44fa-a50f-ffcff69259dd","Type":"ContainerStarted","Data":"d01a5d18f692851d61f12fb70562f4e40b2185b087072645008f6de863013d8d"} Oct 30 22:02:15.254631 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:15.254567 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" podStartSLOduration=2.254553673 podStartE2EDuration="2.254553673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:15.252666805 +0000 UTC m=+1443.843952292" watchObservedRunningTime="2026-10-30 22:02:15.254553673 +0000 UTC m=+1443.845839160" Oct 30 22:02:15.949224 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:15.949192 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:15.951006 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:15.950974 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:15.951172 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:15.951034 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:02:16.948985 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:16.948950 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:16.949493 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:16.949011 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:02:17.949276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:17.949219 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:17.949802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:17.949287 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:02:18.949291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:18.949250 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:18.949805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:18.949315 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:02:19.815094 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:02:19.815034 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:02:19.949648 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:19.949612 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:19.950037 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:19.949666 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:02:20.948802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:20.948694 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 22:02:20.948802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:20.948776 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 22:02:21.953725 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:21.953689 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:21.954221 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:21.954094 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:21.954815 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:21.954795 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:02:22.069745 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:22.069718 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:02:25.632138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.602318 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-lpfj5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:02:25.632138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.627514 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:02:25.632138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.602396 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" probeResult="failure" output="Get \"https://10.134.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:02:25.661844 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.661203 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl" podUID="35879e2e-b0ac-4f01-ab06-79ace61c7f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.34:8000/ready\": context deadline exceeded" Oct 30 22:02:25.756436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.734315 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:02:25.756436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.734374 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:02:25.819466 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.810700 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k"] Oct 30 22:02:25.845109 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.845075 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:25.901235 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.901113 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2d2f63-dockercfg-bbz47\"" Oct 30 22:02:25.901609 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.901579 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2d2f63\"" Oct 30 22:02:25.927556 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.920384 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k"] Oct 30 22:02:25.985435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.984335 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt86q\" (UniqueName: \"kubernetes.io/projected/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-kube-api-access-dt86q\") pod \"payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k\" (UID: \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:25.985435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:25.984403 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k\" (UID: \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:26.090440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:26.089464 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt86q\" (UniqueName: \"kubernetes.io/projected/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-kube-api-access-dt86q\") pod \"payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k\" (UID: \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:26.090440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:26.089540 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k\" (UID: \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:26.105921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:26.099223 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k\" (UID: \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:26.150232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:26.150202 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt86q\" (UniqueName: \"kubernetes.io/projected/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-kube-api-access-dt86q\") pod \"payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k\" (UID: \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:26.294253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:26.294218 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:26.666107 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:26.666080 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k"] Oct 30 22:02:26.668052 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:02:26.668017 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb4e4a20_961b_435f_9a2a_2c5fd80bc432.slice/crio-da45e996423f0c9ca360fcacc06c82b0cb1a477af02b6b45fa34c0f864ffc0d7 WatchSource:0}: Error finding container da45e996423f0c9ca360fcacc06c82b0cb1a477af02b6b45fa34c0f864ffc0d7: Status 404 returned error can't find the container with id da45e996423f0c9ca360fcacc06c82b0cb1a477af02b6b45fa34c0f864ffc0d7 Oct 30 22:02:27.399967 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:27.399918 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" event={"ID":"cb4e4a20-961b-435f-9a2a-2c5fd80bc432","Type":"ContainerStarted","Data":"a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204"} Oct 30 22:02:27.400170 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:27.399994 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:27.400170 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:27.400014 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" event={"ID":"cb4e4a20-961b-435f-9a2a-2c5fd80bc432","Type":"ContainerStarted","Data":"da45e996423f0c9ca360fcacc06c82b0cb1a477af02b6b45fa34c0f864ffc0d7"} Oct 30 22:02:27.511140 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:27.511078 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" podStartSLOduration=2.511059146 podStartE2EDuration="2.511059146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:27.508627385 +0000 UTC m=+1456.099912882" watchObservedRunningTime="2026-10-30 22:02:27.511059146 +0000 UTC m=+1456.102344636" Oct 30 22:02:38.408521 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:38.408491 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:02:43.458434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.458389 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq"] Oct 30 22:02:43.463091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.463065 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.512243 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.512201 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:02:43.512243 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.512201 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:02:43.512477 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.512263 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:02:43.512477 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.512341 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-66p9x\"" Oct 30 22:02:43.522970 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.522944 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b72f34-d257-4fe1-9c9b-6ecab47534bc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.523084 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.523068 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.523137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.523124 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.523173 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.523149 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.523217 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.523170 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.523217 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.523205 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfjxz\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-kube-api-access-gfjxz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.523380 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.523368 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.562987 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.562957 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq"] Oct 30 22:02:43.625124 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.625087 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.625320 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.625174 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b72f34-d257-4fe1-9c9b-6ecab47534bc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.625320 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.625292 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.625478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.625355 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.625478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.625383 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.625583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.625506 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.625583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.625549 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfjxz\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-kube-api-access-gfjxz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.626191 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.626163 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.627526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.627501 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b72f34-d257-4fe1-9c9b-6ecab47534bc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.627832 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.627802 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.628125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.628105 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.628125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.628117 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.686944 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.686910 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.687137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.687114 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfjxz\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-kube-api-access-gfjxz\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bddq\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:43.775968 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:43.775936 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:44.196098 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:44.196065 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq"] Oct 30 22:02:44.197492 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:02:44.197457 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b72f34_d257_4fe1_9c9b_6ecab47534bc.slice/crio-61b256ca263fbafdbb26bca13d2e79e5433e835283e691305d9bde8ce3af014e WatchSource:0}: Error finding container 61b256ca263fbafdbb26bca13d2e79e5433e835283e691305d9bde8ce3af014e: Status 404 returned error can't find the container with id 61b256ca263fbafdbb26bca13d2e79e5433e835283e691305d9bde8ce3af014e Oct 30 22:02:44.200214 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:44.200180 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:44.200313 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:44.200241 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:02:44.506073 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:44.506032 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" event={"ID":"92b72f34-d257-4fe1-9c9b-6ecab47534bc","Type":"ContainerStarted","Data":"8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3"} Oct 30 22:02:44.506622 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:44.506101 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:44.506622 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:44.506120 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" event={"ID":"92b72f34-d257-4fe1-9c9b-6ecab47534bc","Type":"ContainerStarted","Data":"61b256ca263fbafdbb26bca13d2e79e5433e835283e691305d9bde8ce3af014e"} Oct 30 22:02:44.624095 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:44.624043 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" podStartSLOduration=1.624028016 podStartE2EDuration="1.624028016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:44.620688314 +0000 UTC m=+1473.211973800" watchObservedRunningTime="2026-10-30 22:02:44.624028016 +0000 UTC m=+1473.215313502" Oct 30 22:02:45.522999 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:45.522961 5139 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6bddq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:02:45.523595 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:45.523556 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" podUID="92b72f34-d257-4fe1-9c9b-6ecab47534bc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:02:48.517482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:02:48.517450 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:02:49.805709 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:02:49.805664 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:03:11.758508 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:03:11.758477 5139 scope.go:122] "RemoveContainer" containerID="8c8ae9af399ab586fb60efa84255fdd1023905476d61599c99643a839e1935be" Oct 30 22:03:19.804937 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:03:19.804898 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:03:49.806226 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:03:49.806189 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:04:02.694258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.694172 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq"] Oct 30 22:04:02.698615 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.698591 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.796833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.796796 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f9b7e7-serving-cert\"" Oct 30 22:04:02.799833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.799799 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq"] Oct 30 22:04:02.835871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.835836 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-metrics-certs\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.836335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.835979 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6czd7\" (UniqueName: \"kubernetes.io/projected/d200a103-ab8c-45ef-ac71-090c6e420d4f-kube-api-access-6czd7\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.836335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.836044 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-maas-api-tls\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.836335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.836078 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d200a103-ab8c-45ef-ac71-090c6e420d4f-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.937729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.937690 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-metrics-certs\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.937933 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.937753 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6czd7\" (UniqueName: \"kubernetes.io/projected/d200a103-ab8c-45ef-ac71-090c6e420d4f-kube-api-access-6czd7\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.937933 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.937784 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-maas-api-tls\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.937933 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.937805 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d200a103-ab8c-45ef-ac71-090c6e420d4f-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.938557 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.938527 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d200a103-ab8c-45ef-ac71-090c6e420d4f-openshift-service-ca\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.940301 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.940279 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-metrics-certs\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:02.940411 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:02.940328 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-maas-api-tls\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:03.001930 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:03.001897 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6czd7\" (UniqueName: \"kubernetes.io/projected/d200a103-ab8c-45ef-ac71-090c6e420d4f-kube-api-access-6czd7\") pod \"maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:03.011887 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:03.011852 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:03.390166 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:03.390054 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq"] Oct 30 22:04:03.400560 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:04:03.400517 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd200a103_ab8c_45ef_ac71_090c6e420d4f.slice/crio-344f7bae5a638300289a4a7e84d57f1c741bde22e5a57c41ba77091ea53ca628 WatchSource:0}: Error finding container 344f7bae5a638300289a4a7e84d57f1c741bde22e5a57c41ba77091ea53ca628: Status 404 returned error can't find the container with id 344f7bae5a638300289a4a7e84d57f1c741bde22e5a57c41ba77091ea53ca628 Oct 30 22:04:04.053496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:04.053457 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" event={"ID":"d200a103-ab8c-45ef-ac71-090c6e420d4f","Type":"ContainerStarted","Data":"344f7bae5a638300289a4a7e84d57f1c741bde22e5a57c41ba77091ea53ca628"} Oct 30 22:04:05.061083 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:05.061047 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" event={"ID":"d200a103-ab8c-45ef-ac71-090c6e420d4f","Type":"ContainerStarted","Data":"3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b"} Oct 30 22:04:05.061594 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:05.061143 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:05.173360 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:05.173235 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" podStartSLOduration=2.225095063 podStartE2EDuration="3.173217228s" totalImagesPullingTime="948.122165ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:05.172614747 +0000 UTC m=+1553.763900234" watchObservedRunningTime="2026-10-30 22:04:05.173217228 +0000 UTC m=+1553.764502715" Oct 30 22:04:11.073833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:11.073805 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:04:19.807200 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:04:19.807160 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:04:49.806196 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:04:49.806157 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:04:50.981699 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:50.981663 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq"] Oct 30 22:04:50.986178 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:50.986156 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.023009 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.022977 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cd2838-6854-4690-97ff-06e1bb784f68-openshift-service-ca\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.023150 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.023038 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.023150 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.023060 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhlm\" (UniqueName: \"kubernetes.io/projected/c5cd2838-6854-4690-97ff-06e1bb784f68-kube-api-access-7dhlm\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.023150 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.023085 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-metrics-certs\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.037154 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.037129 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-82b62e-serving-cert\"" Oct 30 22:04:51.085986 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.085954 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq"] Oct 30 22:04:51.124487 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.124459 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cd2838-6854-4690-97ff-06e1bb784f68-openshift-service-ca\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.124669 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.124516 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.124669 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.124540 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhlm\" (UniqueName: \"kubernetes.io/projected/c5cd2838-6854-4690-97ff-06e1bb784f68-kube-api-access-7dhlm\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.124669 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.124570 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-metrics-certs\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.124877 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:04:51.124679 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-serving-cert: secret "maas-api-e2e-api-b-82b62e-serving-cert" not found Oct 30 22:04:51.124877 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:04:51.124754 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls podName:c5cd2838-6854-4690-97ff-06e1bb784f68 nodeName:}" failed. No retries permitted until 2026-10-30 22:04:51.624733434 +0000 UTC m=+1600.216018915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls") pod "maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" (UID: "c5cd2838-6854-4690-97ff-06e1bb784f68") : secret "maas-api-e2e-api-b-82b62e-serving-cert" not found Oct 30 22:04:51.125243 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.125215 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cd2838-6854-4690-97ff-06e1bb784f68-openshift-service-ca\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.127121 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.127099 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-metrics-certs\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.184175 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.184140 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhlm\" (UniqueName: \"kubernetes.io/projected/c5cd2838-6854-4690-97ff-06e1bb784f68-kube-api-access-7dhlm\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.633067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.633019 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.635603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.635570 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls\") pod \"maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:51.899820 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:51.899728 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:52.299274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:52.299238 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq"] Oct 30 22:04:52.304327 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:04:52.304296 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5cd2838_6854_4690_97ff_06e1bb784f68.slice/crio-7e0fabe1858b92ed9e007baad991dcf2de52a72773302b33f66e9cbdbc2947bc WatchSource:0}: Error finding container 7e0fabe1858b92ed9e007baad991dcf2de52a72773302b33f66e9cbdbc2947bc: Status 404 returned error can't find the container with id 7e0fabe1858b92ed9e007baad991dcf2de52a72773302b33f66e9cbdbc2947bc Oct 30 22:04:52.350324 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:52.350291 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" event={"ID":"c5cd2838-6854-4690-97ff-06e1bb784f68","Type":"ContainerStarted","Data":"7e0fabe1858b92ed9e007baad991dcf2de52a72773302b33f66e9cbdbc2947bc"} Oct 30 22:04:52.440650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:52.440610 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" podUID="3ab1a8fb7b394ed5c2ad3173d5294ab9" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 22:04:53.359158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:53.359124 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" event={"ID":"c5cd2838-6854-4690-97ff-06e1bb784f68","Type":"ContainerStarted","Data":"db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa"} Oct 30 22:04:53.359570 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:53.359189 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:04:53.474264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:53.474165 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" podStartSLOduration=2.643791237 podStartE2EDuration="3.47415124s" totalImagesPullingTime="830.360003ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:53.471013463 +0000 UTC m=+1602.062298950" watchObservedRunningTime="2026-10-30 22:04:53.47415124 +0000 UTC m=+1602.065436753" Oct 30 22:04:58.042756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:58.042714 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p"] Oct 30 22:04:58.043187 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:58.042942 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" containerID="cri-o://0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe" gracePeriod=30 Oct 30 22:04:59.367699 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:04:59.367654 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.80:8443/health\": dial tcp 10.134.0.80:8443: connect: connection refused" Oct 30 22:05:03.030871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.030832 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:05:03.031296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.030901 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:05:03.358315 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.358286 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 22:05:03.419745 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.419716 5139 generic.go:356] "Generic (PLEG): container finished" podID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerID="db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa" exitCode=1 Oct 30 22:05:03.419914 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.419790 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" event={"ID":"c5cd2838-6854-4690-97ff-06e1bb784f68","Type":"ContainerDied","Data":"db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa"} Oct 30 22:05:03.420306 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.420283 5139 scope.go:122] "RemoveContainer" containerID="db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa" Oct 30 22:05:03.421395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.421366 5139 generic.go:356] "Generic (PLEG): container finished" podID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerID="0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe" exitCode=0 Oct 30 22:05:03.421553 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.421432 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" Oct 30 22:05:03.421553 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.421453 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" event={"ID":"ae2ab003-110a-40b8-b0bc-a86f7f334772","Type":"ContainerDied","Data":"0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe"} Oct 30 22:05:03.421553 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.421514 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p" event={"ID":"ae2ab003-110a-40b8-b0bc-a86f7f334772","Type":"ContainerDied","Data":"b9aa30e72f5e4e9eec66bb9395ae8b912d0b4111830b1dfadfa7a3475ed48954"} Oct 30 22:05:03.421553 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.421545 5139 scope.go:122] "RemoveContainer" containerID="0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe" Oct 30 22:05:03.431580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.431552 5139 scope.go:122] "RemoveContainer" containerID="0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe" Oct 30 22:05:03.431856 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:05:03.431836 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe\": container with ID starting with 0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe not found: ID does not exist" containerID="0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe" Oct 30 22:05:03.431915 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.431869 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe"} err="failed to get container status \"0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe\": rpc error: code = NotFound desc = could not find container \"0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe\": container with ID starting with 0a091421c300ca95fd073cd95263d09a1d2a5b4999fe3c7e4fd7545f4c3c6cbe not found: ID does not exist" Oct 30 22:05:03.494303 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494268 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-socket\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.494549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494357 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-podinfo\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.494549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494401 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-envoy\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.494549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494460 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-credential-socket\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.494549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494495 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-certs\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.494772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494546 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-socket" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.494772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494704 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-token\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-token\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.494772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494748 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-kube-api-access-8d6pz\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-kube-api-access-8d6pz\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.494924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494800 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-data\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.494924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494869 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-certs" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.494924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.494896 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2ab003-110a-40b8-b0bc-a86f7f334772-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2ab003-110a-40b8-b0bc-a86f7f334772-istiod-ca-cert\") pod \"ae2ab003-110a-40b8-b0bc-a86f7f334772\" (UID: \"ae2ab003-110a-40b8-b0bc-a86f7f334772\") " Oct 30 22:05:03.495192 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.495160 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-data" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.495519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.495495 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2ab003-110a-40b8-b0bc-a86f7f334772-istiod-ca-cert" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:03.495688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.495667 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-credential-socket" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.496619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.496596 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.496827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.496622 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae2ab003-110a-40b8-b0bc-a86f7f334772-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.496827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.496640 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.496827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.496656 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.496827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.496668 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.497194 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.497172 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-token" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:03.497305 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.497211 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-podinfo" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:05:03.497859 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.497834 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-kube-api-access-8d6pz" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "kube-api-access-8d6pz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:03.498109 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.498089 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-envoy" pod "ae2ab003-110a-40b8-b0bc-a86f7f334772" (UID: "ae2ab003-110a-40b8-b0bc-a86f7f334772"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:03.597912 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.597874 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.597912 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.597914 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.598232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.597929 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.598232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.597944 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d6pz\" (UniqueName: \"kubernetes.io/projected/ae2ab003-110a-40b8-b0bc-a86f7f334772-kube-api-access-8d6pz\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:03.838796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.838680 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p"] Oct 30 22:05:03.842440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:03.842398 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-fkp6p"] Oct 30 22:05:05.437114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:05.437077 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" event={"ID":"c5cd2838-6854-4690-97ff-06e1bb784f68","Type":"ContainerStarted","Data":"05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910"} Oct 30 22:05:05.437555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:05.437318 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:05:05.544623 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:05.544589 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" path="/var/lib/kubelet/pods/ae2ab003-110a-40b8-b0bc-a86f7f334772/volumes" Oct 30 22:05:10.156270 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.156235 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w"] Oct 30 22:05:10.156832 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.156513 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" podUID="8bb8feac-77b9-47a9-bf58-8c815ca853a4" containerName="maas-api" containerID="cri-o://2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3" gracePeriod=30 Oct 30 22:05:10.444732 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.444703 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:05:10.468192 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.468159 5139 generic.go:356] "Generic (PLEG): container finished" podID="8bb8feac-77b9-47a9-bf58-8c815ca853a4" containerID="2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3" exitCode=0 Oct 30 22:05:10.468358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.468209 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" Oct 30 22:05:10.468358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.468213 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" event={"ID":"8bb8feac-77b9-47a9-bf58-8c815ca853a4","Type":"ContainerDied","Data":"2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3"} Oct 30 22:05:10.468358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.468268 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w" event={"ID":"8bb8feac-77b9-47a9-bf58-8c815ca853a4","Type":"ContainerDied","Data":"e947f93474c7f5dfd6001dccb66f81182edbeca8e5b4b4d56f7982b37e01ec26"} Oct 30 22:05:10.468358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.468298 5139 scope.go:122] "RemoveContainer" containerID="2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3" Oct 30 22:05:10.479106 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.479086 5139 scope.go:122] "RemoveContainer" containerID="2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3" Oct 30 22:05:10.479393 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:05:10.479374 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3\": container with ID starting with 2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3 not found: ID does not exist" containerID="2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3" Oct 30 22:05:10.479469 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.479405 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3"} err="failed to get container status \"2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3\": rpc error: code = NotFound desc = could not find container \"2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3\": container with ID starting with 2a6b8f3186719dc7e5dbb35a65dd564ea1a9481df8d80eb2612dbb0234758bc3 not found: ID does not exist" Oct 30 22:05:10.508916 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.508878 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-metrics-certs\") pod \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " Oct 30 22:05:10.509115 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.508962 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb8feac-77b9-47a9-bf58-8c815ca853a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8feac-77b9-47a9-bf58-8c815ca853a4-openshift-service-ca\") pod \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " Oct 30 22:05:10.509115 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.509082 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb8feac-77b9-47a9-bf58-8c815ca853a4-kube-api-access-7xwqk\" (UniqueName: \"kubernetes.io/projected/8bb8feac-77b9-47a9-bf58-8c815ca853a4-kube-api-access-7xwqk\") pod \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " Oct 30 22:05:10.509691 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.509325 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls\") pod \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\" (UID: \"8bb8feac-77b9-47a9-bf58-8c815ca853a4\") " Oct 30 22:05:10.509691 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.509391 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb8feac-77b9-47a9-bf58-8c815ca853a4-openshift-service-ca" pod "8bb8feac-77b9-47a9-bf58-8c815ca853a4" (UID: "8bb8feac-77b9-47a9-bf58-8c815ca853a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:10.510204 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.510178 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8bb8feac-77b9-47a9-bf58-8c815ca853a4-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:10.511798 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.511769 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb8feac-77b9-47a9-bf58-8c815ca853a4-kube-api-access-7xwqk" pod "8bb8feac-77b9-47a9-bf58-8c815ca853a4" (UID: "8bb8feac-77b9-47a9-bf58-8c815ca853a4"). InnerVolumeSpecName "kube-api-access-7xwqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:10.511956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.511822 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-metrics-certs" pod "8bb8feac-77b9-47a9-bf58-8c815ca853a4" (UID: "8bb8feac-77b9-47a9-bf58-8c815ca853a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:10.511956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.511930 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls" pod "8bb8feac-77b9-47a9-bf58-8c815ca853a4" (UID: "8bb8feac-77b9-47a9-bf58-8c815ca853a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:05:10.615776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.615506 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xwqk\" (UniqueName: \"kubernetes.io/projected/8bb8feac-77b9-47a9-bf58-8c815ca853a4-kube-api-access-7xwqk\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:10.615776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.615541 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:10.615776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.615559 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8bb8feac-77b9-47a9-bf58-8c815ca853a4-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:10.895390 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.895354 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w"] Oct 30 22:05:10.902552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:10.902519 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-4c260505-796c8fbff9-x6d7w"] Oct 30 22:05:11.448602 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:11.448574 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:05:11.543937 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:11.543905 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb8feac-77b9-47a9-bf58-8c815ca853a4" path="/var/lib/kubelet/pods/8bb8feac-77b9-47a9-bf58-8c815ca853a4/volumes" Oct 30 22:05:19.807941 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:05:19.807898 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:05:30.512093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:30.512011 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd"] Oct 30 22:05:30.512729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:30.512315 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" containerID="cri-o://007a5f147c9273c701fb432205b59e2be13340e3b3c06e71f4dd579c1a6df976" gracePeriod=30 Oct 30 22:05:35.646573 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.646541 5139 generic.go:356] "Generic (PLEG): container finished" podID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerID="007a5f147c9273c701fb432205b59e2be13340e3b3c06e71f4dd579c1a6df976" exitCode=0 Oct 30 22:05:35.646929 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.646584 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" event={"ID":"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a","Type":"ContainerDied","Data":"007a5f147c9273c701fb432205b59e2be13340e3b3c06e71f4dd579c1a6df976"} Oct 30 22:05:35.785288 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.785213 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp"] Oct 30 22:05:35.786323 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.786294 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" Oct 30 22:05:35.786323 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.786322 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" Oct 30 22:05:35.786540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.786359 5139 cpu_manager.go:414] "containerMap: removing container" podUID="8bb8feac-77b9-47a9-bf58-8c815ca853a4" containerName="maas-api" Oct 30 22:05:35.786540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.786365 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb8feac-77b9-47a9-bf58-8c815ca853a4" containerName="maas-api" Oct 30 22:05:35.786540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.786523 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2ab003-110a-40b8-b0bc-a86f7f334772" containerName="istio-proxy" Oct 30 22:05:35.786540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.786535 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb8feac-77b9-47a9-bf58-8c815ca853a4" containerName="maas-api" Oct 30 22:05:35.790747 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.790725 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.794164 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.794143 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 22:05:35.858184 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.858148 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-a5f7b9-openshift-default-dockercfg-8tmpg\"" Oct 30 22:05:35.878323 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878295 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-credential-socket\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.878524 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878393 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-socket\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.878524 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878441 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-token\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-token\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.878645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878548 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istiod-ca-cert\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.878645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878631 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-certs\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.878741 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878692 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-data\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.878741 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878696 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-credential-socket" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.878741 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878705 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-socket" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.878899 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878756 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-envoy\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.878899 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878791 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-kube-api-access-zth7k\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-kube-api-access-zth7k\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.878899 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878832 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-podinfo\") pod \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\" (UID: \"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a\") " Oct 30 22:05:35.879060 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878896 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-certs" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.879060 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878913 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-data" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.879060 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.878951 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istiod-ca-cert" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:35.879211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879194 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-data\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.879350 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879320 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-credential-socket\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.879491 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879458 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-istio-token\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.879562 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879514 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd019f2-888d-483b-8d66-51f93f350648-istiod-ca-cert\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.879655 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879629 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-socket\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.879723 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879677 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-envoy\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.879723 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879715 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5w5c\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-kube-api-access-r5w5c\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.879853 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879829 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd019f2-888d-483b-8d66-51f93f350648-istio-podinfo\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.879916 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.879872 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-certs\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.880016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.880000 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.880077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.880023 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.880077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.880039 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.880077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.880054 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.880077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.880068 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.881114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.881080 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-token" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:35.881253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.881208 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-kube-api-access-zth7k" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "kube-api-access-zth7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:35.881253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.881133 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-envoy" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:35.881253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.881205 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-podinfo" pod "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" (UID: "34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:05:35.891011 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.890983 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp"] Oct 30 22:05:35.981344 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981307 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-socket\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981344 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981344 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-envoy\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981642 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981369 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5w5c\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-kube-api-access-r5w5c\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981642 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981397 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd019f2-888d-483b-8d66-51f93f350648-istio-podinfo\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981642 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981565 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-certs\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981727 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-data\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981798 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-credential-socket\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981905 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981877 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-istio-token\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981959 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981917 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd019f2-888d-483b-8d66-51f93f350648-istiod-ca-cert\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.981959 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.981926 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-certs\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.982070 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.982047 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-socket\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.982126 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.982096 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-data\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.982199 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.982134 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.982199 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.982180 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-credential-socket\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.982303 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.982185 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zth7k\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-kube-api-access-zth7k\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.982303 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.982231 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.982303 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.982249 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:35.982646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.982628 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd019f2-888d-483b-8d66-51f93f350648-istiod-ca-cert\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.983816 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.983797 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-envoy\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:35.983913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:35.983896 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd019f2-888d-483b-8d66-51f93f350648-istio-podinfo\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:36.040814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.040732 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-istio-token\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:36.043077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.043050 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5w5c\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-kube-api-access-r5w5c\") pod \"e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:36.103566 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.103527 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:36.274459 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.274396 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb"] Oct 30 22:05:36.275129 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.275110 5139 cpu_manager.go:414] "containerMap: removing container" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" Oct 30 22:05:36.275129 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.275128 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" Oct 30 22:05:36.275288 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.275233 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" containerName="istio-proxy" Oct 30 22:05:36.279190 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.279168 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.330928 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.330848 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4c260505-openshift-default-dockercfg-k4h5b\"" Oct 30 22:05:36.338876 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.338843 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb"] Oct 30 22:05:36.389366 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389331 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.389581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389487 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.389581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389551 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.389673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389584 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972b9d32-22c1-42f8-a368-af1b8838cf47-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.389733 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389712 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.389768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389758 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.389802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389785 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxsbp\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-kube-api-access-cxsbp\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.389932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389917 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.389995 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.389954 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.490932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.490898 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491007 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491072 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491108 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972b9d32-22c1-42f8-a368-af1b8838cf47-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491317 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491149 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491317 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491190 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491317 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491229 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxsbp\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-kube-api-access-cxsbp\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491322 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491361 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491367 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491396 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491778 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491936 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491838 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.491936 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.491884 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972b9d32-22c1-42f8-a368-af1b8838cf47-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.493787 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.493765 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.494160 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.494141 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.548252 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.548223 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp"] Oct 30 22:05:36.551709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.551684 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.552579 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:05:36.552555 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd019f2_888d_483b_8d66_51f93f350648.slice/crio-a1c5f9173b5eb4a065cb3d3dea9bb421af74ba6f64d3c95c100f8eb7d70aa9af WatchSource:0}: Error finding container a1c5f9173b5eb4a065cb3d3dea9bb421af74ba6f64d3c95c100f8eb7d70aa9af: Status 404 returned error can't find the container with id a1c5f9173b5eb4a065cb3d3dea9bb421af74ba6f64d3c95c100f8eb7d70aa9af Oct 30 22:05:36.555289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.555267 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxsbp\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-kube-api-access-cxsbp\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-pnsfb\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.555491 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.555449 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:36.555587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.555538 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:36.555661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.555583 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:36.597405 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.597319 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:36.659343 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.655472 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" event={"ID":"8dd019f2-888d-483b-8d66-51f93f350648","Type":"ContainerStarted","Data":"a1c5f9173b5eb4a065cb3d3dea9bb421af74ba6f64d3c95c100f8eb7d70aa9af"} Oct 30 22:05:36.670382 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.666964 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" event={"ID":"34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a","Type":"ContainerDied","Data":"235c5dfacff1f70f76f15e521acbbc522fd173f715fedfd2efbffc8a1144233e"} Oct 30 22:05:36.670382 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.667040 5139 scope.go:122] "RemoveContainer" containerID="007a5f147c9273c701fb432205b59e2be13340e3b3c06e71f4dd579c1a6df976" Oct 30 22:05:36.670382 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.667223 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd" Oct 30 22:05:36.825628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.825593 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd"] Oct 30 22:05:36.831758 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.831730 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-tsdmd"] Oct 30 22:05:36.988301 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.988215 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb"] Oct 30 22:05:36.992082 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:05:36.992053 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972b9d32_22c1_42f8_a368_af1b8838cf47.slice/crio-447fae7191c4740dd4dfb91e568eae0f35f80400d08b668b9e24d0b876cd2926 WatchSource:0}: Error finding container 447fae7191c4740dd4dfb91e568eae0f35f80400d08b668b9e24d0b876cd2926: Status 404 returned error can't find the container with id 447fae7191c4740dd4dfb91e568eae0f35f80400d08b668b9e24d0b876cd2926 Oct 30 22:05:36.995333 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.995297 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:36.995434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.995374 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:36.995434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:36.995404 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:05:37.545335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:37.545304 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a" path="/var/lib/kubelet/pods/34e3c031-6eef-46bc-a3c8-0ac1ae5ff92a/volumes" Oct 30 22:05:37.738448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:37.738174 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" event={"ID":"8dd019f2-888d-483b-8d66-51f93f350648","Type":"ContainerStarted","Data":"7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968"} Oct 30 22:05:37.760435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:37.759207 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" event={"ID":"972b9d32-22c1-42f8-a368-af1b8838cf47","Type":"ContainerStarted","Data":"a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f"} Oct 30 22:05:37.760435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:37.759310 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" event={"ID":"972b9d32-22c1-42f8-a368-af1b8838cf47","Type":"ContainerStarted","Data":"447fae7191c4740dd4dfb91e568eae0f35f80400d08b668b9e24d0b876cd2926"} Oct 30 22:05:38.047092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:38.047035 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" podStartSLOduration=2.047017177 podStartE2EDuration="2.047017177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:38.045716763 +0000 UTC m=+1646.637002250" watchObservedRunningTime="2026-10-30 22:05:38.047017177 +0000 UTC m=+1646.638302666" Oct 30 22:05:38.047708 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:38.047680 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podStartSLOduration=3.04767222 podStartE2EDuration="3.04767222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:37.919963696 +0000 UTC m=+1646.511249185" watchObservedRunningTime="2026-10-30 22:05:38.04767222 +0000 UTC m=+1646.638957707" Oct 30 22:05:38.110881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:38.110797 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:38.113630 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:38.113387 5139 patch_prober.go:29] interesting pod/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:38.113630 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:38.113469 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:05:38.603442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:38.603391 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:38.603629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:38.603526 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:38.603629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:38.603572 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:05:39.107441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:39.107126 5139 patch_prober.go:29] interesting pod/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:39.107441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:39.107195 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:05:39.614438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:39.614352 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:39.614656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:39.614459 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:05:40.113442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:40.111330 5139 patch_prober.go:29] interesting pod/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:40.113442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:40.111393 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:05:40.604439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:40.600753 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:40.604439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:40.600831 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:05:41.114764 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:41.114730 5139 patch_prober.go:29] interesting pod/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:41.115509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:41.115476 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:05:41.599441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:41.599170 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:41.599441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:41.599235 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:05:42.113113 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:42.112932 5139 patch_prober.go:29] interesting pod/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:42.113113 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:42.113000 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:05:42.598226 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:42.597958 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 22:05:42.598226 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:42.598016 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 22:05:43.107438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:43.106838 5139 patch_prober.go:29] interesting pod/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 30 22:05:43.107438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:43.106899 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 30 22:05:43.675449 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:43.675383 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:43.676922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:43.676838 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:43.677722 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:43.677700 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:05:44.109029 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:44.109001 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:44.109470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:44.109450 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:44.110378 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:44.110354 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:05:45.632809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:45.632767 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs"] Oct 30 22:05:45.633273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:45.633009 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" containerID="cri-o://346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff" gracePeriod=30 Oct 30 22:05:49.806029 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:05:49.805988 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:05:50.897552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.897495 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 22:05:50.925626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.925593 5139 generic.go:356] "Generic (PLEG): container finished" podID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerID="346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff" exitCode=0 Oct 30 22:05:50.925772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.925642 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" Oct 30 22:05:50.925772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.925679 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" event={"ID":"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed","Type":"ContainerDied","Data":"346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff"} Oct 30 22:05:50.925772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.925737 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs" event={"ID":"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed","Type":"ContainerDied","Data":"3638676d96c5a37ade9110dd330cdcf0ac040581f5e4757502feb85e0494107a"} Oct 30 22:05:50.925772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.925763 5139 scope.go:122] "RemoveContainer" containerID="346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff" Oct 30 22:05:50.928040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928018 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-token\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-token\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928148 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928065 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-socket\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928234 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928221 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-credential-socket\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928293 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928257 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istiod-ca-cert\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928298 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-data\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928428 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928398 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-certs\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928457 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-socket" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:50.928494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928473 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-envoy\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928478 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-credential-socket" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:50.928639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928506 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-kube-api-access-nspwn\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-kube-api-access-nspwn\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928609 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-data" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:50.928639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928629 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-podinfo\") pod \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\" (UID: \"a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed\") " Oct 30 22:05:50.928778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928643 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-certs" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:50.928778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.928695 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istiod-ca-cert" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:05:50.929336 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.929304 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:50.929336 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.929335 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:50.929544 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.929351 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:50.929544 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.929366 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:50.929544 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.929381 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:50.930472 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.930446 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-token" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:50.930782 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.930747 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-podinfo" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:05:50.930851 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.930792 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-kube-api-access-nspwn" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "kube-api-access-nspwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:05:50.930851 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.930818 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-envoy" pod "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" (UID: "a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:05:50.956605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.956581 5139 scope.go:122] "RemoveContainer" containerID="346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff" Oct 30 22:05:50.956911 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:05:50.956885 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff\": container with ID starting with 346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff not found: ID does not exist" containerID="346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff" Oct 30 22:05:50.956970 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:50.956920 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff"} err="failed to get container status \"346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff\": rpc error: code = NotFound desc = could not find container \"346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff\": container with ID starting with 346436eb0f405529b2f0f7aa64945fbc2ab1dba2193c94f5bad8701e2779eeff not found: ID does not exist" Oct 30 22:05:51.031032 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:51.030997 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.031032 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:51.031028 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nspwn\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-kube-api-access-nspwn\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.031238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:51.031041 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.031238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:51.031051 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:05:51.340660 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:51.340624 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs"] Oct 30 22:05:51.346097 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:51.346067 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-4njbs"] Oct 30 22:05:51.544119 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:51.544082 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" path="/var/lib/kubelet/pods/a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed/volumes" Oct 30 22:05:57.395378 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:57.395325 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr"] Oct 30 22:05:57.395917 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:05:57.395659 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" containerID="cri-o://2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5" gracePeriod=30 Oct 30 22:06:02.515116 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.515089 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl"] Oct 30 22:06:02.675408 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.675348 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 22:06:02.796939 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.796903 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-envoy\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797132 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.796972 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-credential-socket\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797132 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797081 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istiod-ca-cert\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797132 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797125 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-podinfo\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797154 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-data\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797187 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-socket\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797217 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-token\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-token\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797263 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-credential-socket" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.797601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797454 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istiod-ca-cert" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:02.797601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797464 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-kube-api-access-spfr7\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-kube-api-access-spfr7\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797500 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-certs\") pod \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\" (UID: \"19fe5a42-bf93-49b0-bb7e-c621ed3b867f\") " Oct 30 22:06:02.797601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797490 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-socket" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.797601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.797556 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-data" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.798146 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.798123 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-certs" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.798498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.798463 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.798610 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.798501 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.798610 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.798518 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.798610 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.798531 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.798610 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.798544 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.799674 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.799640 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-token" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:02.799944 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.799913 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-envoy" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:06:02.799944 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.799931 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-kube-api-access-spfr7" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "kube-api-access-spfr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:02.800134 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.800115 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-podinfo" pod "19fe5a42-bf93-49b0-bb7e-c621ed3b867f" (UID: "19fe5a42-bf93-49b0-bb7e-c621ed3b867f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:06:02.900305 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.900269 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.900305 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.900300 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.900305 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.900312 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spfr7\" (UniqueName: \"kubernetes.io/projected/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-kube-api-access-spfr7\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:02.900649 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:02.900322 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19fe5a42-bf93-49b0-bb7e-c621ed3b867f-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:03.009894 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.009854 5139 generic.go:356] "Generic (PLEG): container finished" podID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerID="2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5" exitCode=0 Oct 30 22:06:03.010081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.009913 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" Oct 30 22:06:03.010081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.009947 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" event={"ID":"19fe5a42-bf93-49b0-bb7e-c621ed3b867f","Type":"ContainerDied","Data":"2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5"} Oct 30 22:06:03.010081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.010009 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr" event={"ID":"19fe5a42-bf93-49b0-bb7e-c621ed3b867f","Type":"ContainerDied","Data":"5b20094167df12ef952b74e1c50e9419d2b34e021ca7cf3639b59922e79ab87c"} Oct 30 22:06:03.010081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.010038 5139 scope.go:122] "RemoveContainer" containerID="2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5" Oct 30 22:06:03.020671 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.020648 5139 scope.go:122] "RemoveContainer" containerID="2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5" Oct 30 22:06:03.020965 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:06:03.020945 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5\": container with ID starting with 2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5 not found: ID does not exist" containerID="2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5" Oct 30 22:06:03.021012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.020974 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5"} err="failed to get container status \"2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5\": rpc error: code = NotFound desc = could not find container \"2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5\": container with ID starting with 2b391e47b889318d71957f70cc1c618cb397ddecb15a86ed7d9d9eb5eef28ce5 not found: ID does not exist" Oct 30 22:06:03.171117 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.171086 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr"] Oct 30 22:06:03.178981 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.178943 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-22jlr"] Oct 30 22:06:03.544602 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:03.544571 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" path="/var/lib/kubelet/pods/19fe5a42-bf93-49b0-bb7e-c621ed3b867f/volumes" Oct 30 22:06:07.516455 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:07.516389 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" podUID="01451298-22d3-48eb-a464-8ff1c7b32a31" containerName="payload-processing" containerID="cri-o://2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463" gracePeriod=25 Oct 30 22:06:07.790527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:07.790468 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:06:07.968727 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:07.968695 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01451298-22d3-48eb-a464-8ff1c7b32a31-kube-api-access-cw8q6\" (UniqueName: \"kubernetes.io/projected/01451298-22d3-48eb-a464-8ff1c7b32a31-kube-api-access-cw8q6\") pod \"01451298-22d3-48eb-a464-8ff1c7b32a31\" (UID: \"01451298-22d3-48eb-a464-8ff1c7b32a31\") " Oct 30 22:06:07.968904 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:07.968890 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01451298-22d3-48eb-a464-8ff1c7b32a31-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01451298-22d3-48eb-a464-8ff1c7b32a31-plugins-config-volume\") pod \"01451298-22d3-48eb-a464-8ff1c7b32a31\" (UID: \"01451298-22d3-48eb-a464-8ff1c7b32a31\") " Oct 30 22:06:07.969278 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:07.969247 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01451298-22d3-48eb-a464-8ff1c7b32a31-plugins-config-volume" pod "01451298-22d3-48eb-a464-8ff1c7b32a31" (UID: "01451298-22d3-48eb-a464-8ff1c7b32a31"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:07.970785 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:07.970764 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01451298-22d3-48eb-a464-8ff1c7b32a31-kube-api-access-cw8q6" pod "01451298-22d3-48eb-a464-8ff1c7b32a31" (UID: "01451298-22d3-48eb-a464-8ff1c7b32a31"). InnerVolumeSpecName "kube-api-access-cw8q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:08.042923 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.042532 5139 generic.go:356] "Generic (PLEG): container finished" podID="01451298-22d3-48eb-a464-8ff1c7b32a31" containerID="2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463" exitCode=0 Oct 30 22:06:08.042923 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.042591 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" Oct 30 22:06:08.042923 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.042771 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" event={"ID":"01451298-22d3-48eb-a464-8ff1c7b32a31","Type":"ContainerDied","Data":"2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463"} Oct 30 22:06:08.042923 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.042820 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl" event={"ID":"01451298-22d3-48eb-a464-8ff1c7b32a31","Type":"ContainerDied","Data":"359ba8ddd37fcbad8061b60e508a423aaaa5ddebcc1821e6b60276d68c03f34f"} Oct 30 22:06:08.042923 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.042849 5139 scope.go:122] "RemoveContainer" containerID="2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463" Oct 30 22:06:08.055676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.055654 5139 scope.go:122] "RemoveContainer" containerID="2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463" Oct 30 22:06:08.056008 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:06:08.055989 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463\": container with ID starting with 2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463 not found: ID does not exist" containerID="2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463" Oct 30 22:06:08.056088 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.056022 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463"} err="failed to get container status \"2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463\": rpc error: code = NotFound desc = could not find container \"2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463\": container with ID starting with 2533ba3a6c3ae6ea593df62b0afb0cfc4f5e8ab23b06ec8737c52dbc087ff463 not found: ID does not exist" Oct 30 22:06:08.070587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.070562 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01451298-22d3-48eb-a464-8ff1c7b32a31-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:08.070587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.070584 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cw8q6\" (UniqueName: \"kubernetes.io/projected/01451298-22d3-48eb-a464-8ff1c7b32a31-kube-api-access-cw8q6\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:08.199185 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.199149 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl"] Oct 30 22:06:08.202220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:08.202192 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-695f5c-7698569c67-425hl"] Oct 30 22:06:09.544273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:09.544242 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01451298-22d3-48eb-a464-8ff1c7b32a31" path="/var/lib/kubelet/pods/01451298-22d3-48eb-a464-8ff1c7b32a31/volumes" Oct 30 22:06:18.736753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:18.736718 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k"] Oct 30 22:06:19.806320 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:06:19.806280 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:06:24.107994 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:24.107952 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" podUID="cb4e4a20-961b-435f-9a2a-2c5fd80bc432" containerName="payload-pre-processing" containerID="cri-o://a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204" gracePeriod=25 Oct 30 22:06:24.394922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:24.394896 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:06:24.514714 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:24.514677 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-kube-api-access-dt86q\" (UniqueName: \"kubernetes.io/projected/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-kube-api-access-dt86q\") pod \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\" (UID: \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\") " Oct 30 22:06:24.514910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:24.514808 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-plugins-config-volume\") pod \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\" (UID: \"cb4e4a20-961b-435f-9a2a-2c5fd80bc432\") " Oct 30 22:06:24.515371 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:24.515339 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-plugins-config-volume" pod "cb4e4a20-961b-435f-9a2a-2c5fd80bc432" (UID: "cb4e4a20-961b-435f-9a2a-2c5fd80bc432"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:06:24.529063 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:24.529032 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-kube-api-access-dt86q" pod "cb4e4a20-961b-435f-9a2a-2c5fd80bc432" (UID: "cb4e4a20-961b-435f-9a2a-2c5fd80bc432"). InnerVolumeSpecName "kube-api-access-dt86q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:06:24.617125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:24.617041 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt86q\" (UniqueName: \"kubernetes.io/projected/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-kube-api-access-dt86q\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:24.617125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:24.617069 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb4e4a20-961b-435f-9a2a-2c5fd80bc432-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:06:25.145584 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.145540 5139 generic.go:356] "Generic (PLEG): container finished" podID="cb4e4a20-961b-435f-9a2a-2c5fd80bc432" containerID="a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204" exitCode=0 Oct 30 22:06:25.145999 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.145614 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" Oct 30 22:06:25.145999 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.145614 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" event={"ID":"cb4e4a20-961b-435f-9a2a-2c5fd80bc432","Type":"ContainerDied","Data":"a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204"} Oct 30 22:06:25.145999 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.145696 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k" event={"ID":"cb4e4a20-961b-435f-9a2a-2c5fd80bc432","Type":"ContainerDied","Data":"da45e996423f0c9ca360fcacc06c82b0cb1a477af02b6b45fa34c0f864ffc0d7"} Oct 30 22:06:25.145999 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.145727 5139 scope.go:122] "RemoveContainer" containerID="a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204" Oct 30 22:06:25.163528 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.163504 5139 scope.go:122] "RemoveContainer" containerID="a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204" Oct 30 22:06:25.163841 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:06:25.163820 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204\": container with ID starting with a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204 not found: ID does not exist" containerID="a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204" Oct 30 22:06:25.163916 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.163849 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204"} err="failed to get container status \"a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204\": rpc error: code = NotFound desc = could not find container \"a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204\": container with ID starting with a7bdd0a7cd0d4af649e29d08efc7fa45c0ed54673fd3077adf68b691ddd51204 not found: ID does not exist" Oct 30 22:06:25.300307 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.300273 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k"] Oct 30 22:06:25.347427 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.347381 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-2d2f63-8659869dff-n9h9k"] Oct 30 22:06:25.544944 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:25.544911 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb4e4a20-961b-435f-9a2a-2c5fd80bc432" path="/var/lib/kubelet/pods/cb4e4a20-961b-435f-9a2a-2c5fd80bc432/volumes" Oct 30 22:06:49.805347 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:06:49.805310 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:06:51.880821 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.880736 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh"] Oct 30 22:06:51.881332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881315 5139 cpu_manager.go:414] "containerMap: removing container" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" Oct 30 22:06:51.881374 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881334 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" Oct 30 22:06:51.881374 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881346 5139 cpu_manager.go:414] "containerMap: removing container" podUID="01451298-22d3-48eb-a464-8ff1c7b32a31" containerName="payload-processing" Oct 30 22:06:51.881374 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881352 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01451298-22d3-48eb-a464-8ff1c7b32a31" containerName="payload-processing" Oct 30 22:06:51.881374 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881360 5139 cpu_manager.go:414] "containerMap: removing container" podUID="cb4e4a20-961b-435f-9a2a-2c5fd80bc432" containerName="payload-pre-processing" Oct 30 22:06:51.881374 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881366 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb4e4a20-961b-435f-9a2a-2c5fd80bc432" containerName="payload-pre-processing" Oct 30 22:06:51.881545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881392 5139 cpu_manager.go:414] "containerMap: removing container" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" Oct 30 22:06:51.881545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881397 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" Oct 30 22:06:51.881545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881525 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fe5a42-bf93-49b0-bb7e-c621ed3b867f" containerName="istio-proxy" Oct 30 22:06:51.881545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881540 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="01451298-22d3-48eb-a464-8ff1c7b32a31" containerName="payload-processing" Oct 30 22:06:51.881675 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881547 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb4e4a20-961b-435f-9a2a-2c5fd80bc432" containerName="payload-pre-processing" Oct 30 22:06:51.881675 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.881558 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b29d4b-e714-4f0e-ba9e-e3cc1806c7ed" containerName="istio-proxy" Oct 30 22:06:51.885178 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.885159 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:51.975855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:51.975819 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh"] Oct 30 22:06:52.017151 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017120 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.017359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017162 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.017359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017256 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.017359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017342 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d121479a-7398-4a24-bccc-283be7383d19-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.017501 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017392 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.017552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017504 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d121479a-7398-4a24-bccc-283be7383d19-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.017552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017546 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.017624 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017570 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqltv\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-kube-api-access-mqltv\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.017670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.017638 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119357 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119403 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119443 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119469 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d121479a-7398-4a24-bccc-283be7383d19-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119495 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119513 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d121479a-7398-4a24-bccc-283be7383d19-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119542 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119652 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqltv\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-kube-api-access-mqltv\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119954 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119683 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119954 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119848 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.119954 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.119905 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.120112 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.120004 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.120235 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.120216 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.120444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.120407 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d121479a-7398-4a24-bccc-283be7383d19-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.121844 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.121823 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.121952 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.121933 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d121479a-7398-4a24-bccc-283be7383d19-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.177720 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.177642 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.177893 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.177785 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqltv\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-kube-api-access-mqltv\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-72zzh\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.197823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.197791 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:52.627148 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.627109 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh"] Oct 30 22:06:52.648781 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.648732 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:06:52.648973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.648840 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:06:52.648973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:52.648889 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:06:53.309749 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:53.309712 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" event={"ID":"d121479a-7398-4a24-bccc-283be7383d19","Type":"ContainerStarted","Data":"d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22"} Oct 30 22:06:53.310152 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:53.309796 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" event={"ID":"d121479a-7398-4a24-bccc-283be7383d19","Type":"ContainerStarted","Data":"ed52b953cbb9d76f378496b21aef4fbc4f3deee669fe5f77d0c4655c9fea3537"} Oct 30 22:06:53.424780 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:53.424722 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" podStartSLOduration=2.4247083209999998 podStartE2EDuration="2.424708321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:53.424087519 +0000 UTC m=+1722.015373008" watchObservedRunningTime="2026-10-30 22:06:53.424708321 +0000 UTC m=+1722.015993881" Oct 30 22:06:54.198366 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:54.198328 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:54.200111 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:54.200083 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-72zzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:06:54.200228 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:54.200144 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:06:55.198976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:55.198938 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-72zzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:06:55.199442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:55.199008 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:06:56.198397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:56.198362 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-72zzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:06:56.198601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:56.198447 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:06:57.202443 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:57.201802 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-72zzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:06:57.202443 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:57.201868 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:06:58.198247 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:58.198203 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-72zzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 22:06:58.198450 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:58.198289 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 22:06:59.203096 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:59.203059 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:59.203638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:59.203615 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:06:59.204315 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:06:59.204292 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:07:07.008986 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.008941 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp"] Oct 30 22:07:07.013382 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.013357 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.114541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.114502 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp"] Oct 30 22:07:07.156052 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156014 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9txn\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-kube-api-access-h9txn\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.156228 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156199 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.156311 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156294 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.156349 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156322 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.156517 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156493 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.156596 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156529 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.156596 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156548 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.156704 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156682 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.156782 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.156766 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258172 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258127 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258178 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258208 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258250 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258290 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258316 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9txn\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-kube-api-access-h9txn\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258504 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258607 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258648 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258670 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.258918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258731 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.259038 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.258966 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.259136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.259112 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.259407 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.259383 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.260949 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.260928 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.261045 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.261028 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.316148 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.316115 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9txn\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-kube-api-access-h9txn\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.316278 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.316117 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lgjkp\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.328168 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.328143 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:07.756553 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.756515 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp"] Oct 30 22:07:07.761902 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:07:07.761872 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53278e91_ddd0_4dbb_920e_12c2bc1b7658.slice/crio-af09c420067da56af92d27a69567c4898f6fdfd81208cc4d5a2d0744f227e008 WatchSource:0}: Error finding container af09c420067da56af92d27a69567c4898f6fdfd81208cc4d5a2d0744f227e008: Status 404 returned error can't find the container with id af09c420067da56af92d27a69567c4898f6fdfd81208cc4d5a2d0744f227e008 Oct 30 22:07:07.764686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.764650 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:07.764802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.764736 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:07.764802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:07.764782 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:08.412449 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:08.412393 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" event={"ID":"53278e91-ddd0-4dbb-920e-12c2bc1b7658","Type":"ContainerStarted","Data":"11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b"} Oct 30 22:07:08.412955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:08.412483 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" event={"ID":"53278e91-ddd0-4dbb-920e-12c2bc1b7658","Type":"ContainerStarted","Data":"af09c420067da56af92d27a69567c4898f6fdfd81208cc4d5a2d0744f227e008"} Oct 30 22:07:08.625966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:08.625903 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" podStartSLOduration=2.625891191 podStartE2EDuration="2.625891191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:08.59123781 +0000 UTC m=+1737.182523297" watchObservedRunningTime="2026-10-30 22:07:08.625891191 +0000 UTC m=+1737.217176679" Oct 30 22:07:09.329136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:09.329101 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:09.330921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:09.330890 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:07:09.331075 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:09.330961 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:07:10.329360 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:10.329325 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:07:10.329799 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:10.329389 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:07:11.328973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:11.328940 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:07:11.329160 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:11.328995 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:07:12.329509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:12.329471 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:07:12.329939 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:12.329525 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:07:13.329229 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:13.329186 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 30 22:07:13.329440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:13.329264 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 30 22:07:14.338311 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:14.338275 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:14.338821 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:14.338725 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:14.339506 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:14.339485 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:07:19.806604 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:07:19.806555 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:07:22.125287 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.125245 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv"] Oct 30 22:07:22.130843 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.130822 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.227844 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.227804 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.228031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.227922 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.228081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.228054 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.228120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.228097 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.228162 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.228123 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.228232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.228206 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.228411 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.228386 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.228526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.228506 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7rp\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-kube-api-access-mn7rp\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.228633 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.228614 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330210 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330176 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7rp\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-kube-api-access-mn7rp\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330380 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330252 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330380 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330365 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330475 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330402 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330528 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330478 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330528 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330507 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330602 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330581 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330675 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330652 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330807 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330721 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330893 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330855 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.330957 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.330909 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.331057 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.331031 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.331280 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.331260 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.331534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.331510 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.334056 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.334031 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.334262 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.334240 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.336227 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.336184 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv"] Oct 30 22:07:22.388433 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.388331 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.388601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.388544 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7rp\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-kube-api-access-mn7rp\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-98zzv\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.444114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.444081 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:22.873377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.873341 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv"] Oct 30 22:07:22.879873 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:07:22.879844 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a79712_1ddf_4042_8bd3_922bc0e399c6.slice/crio-4b86b499ec71cd0e6676255e158d11be931a76b315312e8ae3ecf9a11d386f7c WatchSource:0}: Error finding container 4b86b499ec71cd0e6676255e158d11be931a76b315312e8ae3ecf9a11d386f7c: Status 404 returned error can't find the container with id 4b86b499ec71cd0e6676255e158d11be931a76b315312e8ae3ecf9a11d386f7c Oct 30 22:07:22.882604 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.882571 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:22.882694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.882638 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:22.882694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:22.882672 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:07:23.522362 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:23.522326 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" event={"ID":"f3a79712-1ddf-4042-8bd3-922bc0e399c6","Type":"ContainerStarted","Data":"6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6"} Oct 30 22:07:23.522853 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:23.522388 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" event={"ID":"f3a79712-1ddf-4042-8bd3-922bc0e399c6","Type":"ContainerStarted","Data":"4b86b499ec71cd0e6676255e158d11be931a76b315312e8ae3ecf9a11d386f7c"} Oct 30 22:07:23.653067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:23.653005 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" podStartSLOduration=1.652986532 podStartE2EDuration="1.652986532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:23.65109545 +0000 UTC m=+1752.242380936" watchObservedRunningTime="2026-10-30 22:07:23.652986532 +0000 UTC m=+1752.244272022" Oct 30 22:07:24.444329 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:24.444288 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:24.446551 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:24.446517 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-98zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:24.446707 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:24.446576 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:07:25.446445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:25.446256 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-98zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:25.446445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:25.446324 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:07:26.445119 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:26.445079 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-98zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:26.445299 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:26.445140 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:07:27.448442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:27.447554 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-98zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:27.448442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:27.447622 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:07:28.445544 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:28.445501 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-98zzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 22:07:28.445760 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:28.445604 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 22:07:29.633821 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:29.633786 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:29.634549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:29.634492 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:29.635313 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:29.635294 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:07:45.404264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:45.404221 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6"] Oct 30 22:07:45.404858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:45.404501 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" containerID="cri-o://aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452" gracePeriod=30 Oct 30 22:07:45.449251 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:45.449217 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd"] Oct 30 22:07:45.449489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:45.449437 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" containerID="cri-o://59c36f7cb7f027b2af63017c3f1cf970dafeb956288aa43141eec537deff2320" gracePeriod=30 Oct 30 22:07:45.452264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:45.452229 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx"] Oct 30 22:07:45.452535 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:45.452495 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" containerID="cri-o://357f9438b4b943694b6bba9b8016000b3ff3f1a272a388359a0ae6a4f17e80ae" gracePeriod=30 Oct 30 22:07:45.495372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:45.495324 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw"] Oct 30 22:07:45.495600 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:45.495576 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" containerID="cri-o://01c80f9382d8e8222d0b6ec48b4cb1c0890a895557b7e8c122e9d29f0df2671e" gracePeriod=30 Oct 30 22:07:48.445334 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:48.445288 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:07:48.445910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:48.445367 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:07:49.837217 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:07:49.837179 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:07:50.748188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.748159 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 22:07:50.781732 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.781688 5139 generic.go:356] "Generic (PLEG): container finished" podID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerID="aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452" exitCode=0 Oct 30 22:07:50.781911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.781769 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" Oct 30 22:07:50.781911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.781780 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" event={"ID":"27903e11-7864-4b21-84a8-17ed8c9ca170","Type":"ContainerDied","Data":"aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452"} Oct 30 22:07:50.782040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.781934 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6" event={"ID":"27903e11-7864-4b21-84a8-17ed8c9ca170","Type":"ContainerDied","Data":"0134883edcaab2cd4bfcce3921d967aa04b260986171908e032d39756930c2a8"} Oct 30 22:07:50.782040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.781990 5139 scope.go:122] "RemoveContainer" containerID="aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452" Oct 30 22:07:50.783978 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.783953 5139 generic.go:356] "Generic (PLEG): container finished" podID="f94b7c7c-9e17-4087-b287-ffe225700146" containerID="59c36f7cb7f027b2af63017c3f1cf970dafeb956288aa43141eec537deff2320" exitCode=0 Oct 30 22:07:50.784113 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.784072 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" event={"ID":"f94b7c7c-9e17-4087-b287-ffe225700146","Type":"ContainerDied","Data":"59c36f7cb7f027b2af63017c3f1cf970dafeb956288aa43141eec537deff2320"} Oct 30 22:07:50.786544 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.786519 5139 generic.go:356] "Generic (PLEG): container finished" podID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerID="01c80f9382d8e8222d0b6ec48b4cb1c0890a895557b7e8c122e9d29f0df2671e" exitCode=0 Oct 30 22:07:50.786679 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.786594 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" event={"ID":"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c","Type":"ContainerDied","Data":"01c80f9382d8e8222d0b6ec48b4cb1c0890a895557b7e8c122e9d29f0df2671e"} Oct 30 22:07:50.789119 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.789021 5139 generic.go:356] "Generic (PLEG): container finished" podID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerID="357f9438b4b943694b6bba9b8016000b3ff3f1a272a388359a0ae6a4f17e80ae" exitCode=0 Oct 30 22:07:50.789119 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.789066 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" event={"ID":"231eef6e-a1f5-44fa-a50f-ffcff69259dd","Type":"ContainerDied","Data":"357f9438b4b943694b6bba9b8016000b3ff3f1a272a388359a0ae6a4f17e80ae"} Oct 30 22:07:50.808645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.802951 5139 scope.go:122] "RemoveContainer" containerID="aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452" Oct 30 22:07:50.808645 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:07:50.803359 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452\": container with ID starting with aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452 not found: ID does not exist" containerID="aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452" Oct 30 22:07:50.808645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.803402 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452"} err="failed to get container status \"aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452\": rpc error: code = NotFound desc = could not find container \"aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452\": container with ID starting with aff285592e55d96d830d70f71f788ca8b35712b7dd85c2e0ef5fd47671240452 not found: ID does not exist" Oct 30 22:07:50.839524 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.839489 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:07:50.851977 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.851956 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 22:07:50.857588 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.857565 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:07:50.869753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.869728 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-kube-api-access-snv8l\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-kube-api-access-snv8l\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.869879 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.869761 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27903e11-7864-4b21-84a8-17ed8c9ca170-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27903e11-7864-4b21-84a8-17ed8c9ca170-istiod-ca-cert\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.869943 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.869927 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-token\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-token\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.870101 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870085 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-podinfo\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.870137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870125 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-credential-socket\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.870183 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870144 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27903e11-7864-4b21-84a8-17ed8c9ca170-istiod-ca-cert" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:50.870233 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870201 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-data\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.870381 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870360 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-credential-socket" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.870500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870406 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-envoy\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.870500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870464 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-certs\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.870637 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870531 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-socket\") pod \"27903e11-7864-4b21-84a8-17ed8c9ca170\" (UID: \"27903e11-7864-4b21-84a8-17ed8c9ca170\") " Oct 30 22:07:50.870637 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870548 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-data" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.870821 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870789 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-certs" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.870988 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.870966 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-socket" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.871236 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.871214 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.871322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.871240 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27903e11-7864-4b21-84a8-17ed8c9ca170-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.871322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.871254 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.871322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.871269 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.871322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.871283 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.872125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.872102 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-token" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:50.872220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.872135 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-kube-api-access-snv8l" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "kube-api-access-snv8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:50.872289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.872269 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-podinfo" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:07:50.872545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.872524 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-envoy" pod "27903e11-7864-4b21-84a8-17ed8c9ca170" (UID: "27903e11-7864-4b21-84a8-17ed8c9ca170"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.972658 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972569 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-podinfo\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.972658 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972615 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istiod-ca-cert\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.972910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972673 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-data\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.972910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972704 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-data\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.972910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972737 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-certs\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.972910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972768 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-envoy\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.972910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972801 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-credential-socket\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.972910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972848 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-token\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-token\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.972910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972888 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istiod-ca-cert\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972935 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f94b7c7c-9e17-4087-b287-ffe225700146-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94b7c7c-9e17-4087-b287-ffe225700146-istio-podinfo\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.972969 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-envoy\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973008 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-socket\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973004 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-data" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973022 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-data" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973058 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-credential-socket\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973095 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istiod-ca-cert" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973110 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-kube-api-access-c2k5t\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-kube-api-access-c2k5t\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973160 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-socket\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.973272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973239 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-podinfo\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973281 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-socket\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973312 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-kube-api-access-fwlln\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-kube-api-access-fwlln\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973347 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-certs\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973377 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-certs\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973449 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-credential-socket\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973476 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-envoy\") pod \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\" (UID: \"231eef6e-a1f5-44fa-a50f-ffcff69259dd\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973504 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-token\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-token\") pod \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\" (UID: \"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973562 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94b7c7c-9e17-4087-b287-ffe225700146-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94b7c7c-9e17-4087-b287-ffe225700146-istiod-ca-cert\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973593 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-data\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973628 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-kube-api-access-plf48\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-kube-api-access-plf48\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.973855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973681 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-istio-token\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-istio-token\") pod \"f94b7c7c-9e17-4087-b287-ffe225700146\" (UID: \"f94b7c7c-9e17-4087-b287-ffe225700146\") " Oct 30 22:07:50.974457 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973108 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-certs" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.974457 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973389 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istiod-ca-cert" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:50.974457 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973714 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-credential-socket" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.974457 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973751 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-credential-socket" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974452 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-socket" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.973907 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-certs" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974064 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-socket" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974436 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974522 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974540 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974589 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27903e11-7864-4b21-84a8-17ed8c9ca170-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974605 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974622 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snv8l\" (UniqueName: \"kubernetes.io/projected/27903e11-7864-4b21-84a8-17ed8c9ca170-kube-api-access-snv8l\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.974673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974639 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:50.975232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974647 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-socket" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.975232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.974881 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-data" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.975232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.975118 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-certs" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.975399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.975291 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94b7c7c-9e17-4087-b287-ffe225700146-istiod-ca-cert" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:07:50.975399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.975337 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-credential-socket" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.975861 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.975803 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-podinfo" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:07:50.975861 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.975835 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f94b7c7c-9e17-4087-b287-ffe225700146-istio-podinfo" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:07:50.976185 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.976161 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-token" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:50.976263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.976251 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-podinfo" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:07:50.977093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.977068 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-istio-token" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:50.977275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.977251 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-kube-api-access-c2k5t" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "kube-api-access-c2k5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:50.977359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.977259 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-envoy" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.977723 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.977699 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-envoy" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.977978 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.977963 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-kube-api-access-plf48" pod "f94b7c7c-9e17-4087-b287-ffe225700146" (UID: "f94b7c7c-9e17-4087-b287-ffe225700146"). InnerVolumeSpecName "kube-api-access-plf48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:50.978358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.978336 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-kube-api-access-fwlln" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "kube-api-access-fwlln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:50.978483 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.978351 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-envoy" pod "231eef6e-a1f5-44fa-a50f-ffcff69259dd" (UID: "231eef6e-a1f5-44fa-a50f-ffcff69259dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:07:50.978766 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:50.978748 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-token" pod "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" (UID: "bf198bb7-20b5-43d9-a2ba-f17d2b19d40c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:07:51.075476 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075442 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f94b7c7c-9e17-4087-b287-ffe225700146-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075476 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075476 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075492 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-plf48\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-kube-api-access-plf48\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075504 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f94b7c7c-9e17-4087-b287-ffe225700146-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075513 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075525 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075537 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075547 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075555 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075563 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075572 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f94b7c7c-9e17-4087-b287-ffe225700146-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075581 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075589 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075597 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075612 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2k5t\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-kube-api-access-c2k5t\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075622 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075630 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075639 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075649 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwlln\" (UniqueName: \"kubernetes.io/projected/231eef6e-a1f5-44fa-a50f-ffcff69259dd-kube-api-access-fwlln\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075657 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075666 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f94b7c7c-9e17-4087-b287-ffe225700146-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075674 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075688 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/231eef6e-a1f5-44fa-a50f-ffcff69259dd-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.075739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.075697 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:07:51.425144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.425109 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6"] Oct 30 22:07:51.433497 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.433158 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-7bkl6"] Oct 30 22:07:51.547972 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.547937 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" path="/var/lib/kubelet/pods/27903e11-7864-4b21-84a8-17ed8c9ca170/volumes" Oct 30 22:07:51.797016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.796986 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" Oct 30 22:07:51.797016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.797003 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd" event={"ID":"f94b7c7c-9e17-4087-b287-ffe225700146","Type":"ContainerDied","Data":"c13277d7b9856f2034fea7bdbe9faee2c268bcb30c203c9f9cf897dca69b7810"} Oct 30 22:07:51.797299 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.797063 5139 scope.go:122] "RemoveContainer" containerID="59c36f7cb7f027b2af63017c3f1cf970dafeb956288aa43141eec537deff2320" Oct 30 22:07:51.798695 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.798670 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" Oct 30 22:07:51.798846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.798698 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw" event={"ID":"bf198bb7-20b5-43d9-a2ba-f17d2b19d40c","Type":"ContainerDied","Data":"9631f85b14457712b1ce36c2ca69dc4a4e8726493143013f5397104363ff3025"} Oct 30 22:07:51.800257 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.800235 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" event={"ID":"231eef6e-a1f5-44fa-a50f-ffcff69259dd","Type":"ContainerDied","Data":"d01a5d18f692851d61f12fb70562f4e40b2185b087072645008f6de863013d8d"} Oct 30 22:07:51.800351 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.800267 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx" Oct 30 22:07:51.807609 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.807594 5139 scope.go:122] "RemoveContainer" containerID="01c80f9382d8e8222d0b6ec48b4cb1c0890a895557b7e8c122e9d29f0df2671e" Oct 30 22:07:51.816496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.816480 5139 scope.go:122] "RemoveContainer" containerID="357f9438b4b943694b6bba9b8016000b3ff3f1a272a388359a0ae6a4f17e80ae" Oct 30 22:07:51.956227 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.956194 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd"] Oct 30 22:07:51.966797 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:51.966769 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-kgdjd"] Oct 30 22:07:52.072955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:52.072860 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx"] Oct 30 22:07:52.077380 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:52.077351 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8m8rx"] Oct 30 22:07:52.179969 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:52.179939 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw"] Oct 30 22:07:52.187557 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:52.187525 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-p84zw"] Oct 30 22:07:53.545581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:53.545546 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" path="/var/lib/kubelet/pods/231eef6e-a1f5-44fa-a50f-ffcff69259dd/volumes" Oct 30 22:07:53.546066 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:53.546051 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" path="/var/lib/kubelet/pods/bf198bb7-20b5-43d9-a2ba-f17d2b19d40c/volumes" Oct 30 22:07:53.546502 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:07:53.546489 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" path="/var/lib/kubelet/pods/f94b7c7c-9e17-4087-b287-ffe225700146/volumes" Oct 30 22:08:19.804839 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:08:19.804799 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:08:49.805891 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:08:49.805855 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:08:53.332198 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.332158 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq"] Oct 30 22:08:53.332677 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.332370 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" podUID="d200a103-ab8c-45ef-ac71-090c6e420d4f" containerName="maas-api" containerID="cri-o://3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b" gracePeriod=30 Oct 30 22:08:53.634836 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.634812 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:08:53.788497 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.788458 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-maas-api-tls\") pod \"d200a103-ab8c-45ef-ac71-090c6e420d4f\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " Oct 30 22:08:53.788722 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.788683 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-metrics-certs\") pod \"d200a103-ab8c-45ef-ac71-090c6e420d4f\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " Oct 30 22:08:53.788815 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.788796 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d200a103-ab8c-45ef-ac71-090c6e420d4f-kube-api-access-6czd7\" (UniqueName: \"kubernetes.io/projected/d200a103-ab8c-45ef-ac71-090c6e420d4f-kube-api-access-6czd7\") pod \"d200a103-ab8c-45ef-ac71-090c6e420d4f\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " Oct 30 22:08:53.788906 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.788889 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d200a103-ab8c-45ef-ac71-090c6e420d4f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d200a103-ab8c-45ef-ac71-090c6e420d4f-openshift-service-ca\") pod \"d200a103-ab8c-45ef-ac71-090c6e420d4f\" (UID: \"d200a103-ab8c-45ef-ac71-090c6e420d4f\") " Oct 30 22:08:53.789843 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.789811 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d200a103-ab8c-45ef-ac71-090c6e420d4f-openshift-service-ca" pod "d200a103-ab8c-45ef-ac71-090c6e420d4f" (UID: "d200a103-ab8c-45ef-ac71-090c6e420d4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:08:53.791509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.791465 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-maas-api-tls" pod "d200a103-ab8c-45ef-ac71-090c6e420d4f" (UID: "d200a103-ab8c-45ef-ac71-090c6e420d4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:53.791684 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.791575 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-metrics-certs" pod "d200a103-ab8c-45ef-ac71-090c6e420d4f" (UID: "d200a103-ab8c-45ef-ac71-090c6e420d4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:08:53.792059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.792030 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d200a103-ab8c-45ef-ac71-090c6e420d4f-kube-api-access-6czd7" pod "d200a103-ab8c-45ef-ac71-090c6e420d4f" (UID: "d200a103-ab8c-45ef-ac71-090c6e420d4f"). InnerVolumeSpecName "kube-api-access-6czd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:08:53.890696 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.890570 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:53.890696 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.890634 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6czd7\" (UniqueName: \"kubernetes.io/projected/d200a103-ab8c-45ef-ac71-090c6e420d4f-kube-api-access-6czd7\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:53.890696 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.890646 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d200a103-ab8c-45ef-ac71-090c6e420d4f-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:53.890696 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:53.890656 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d200a103-ab8c-45ef-ac71-090c6e420d4f-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:08:54.263479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.259109 5139 generic.go:356] "Generic (PLEG): container finished" podID="d200a103-ab8c-45ef-ac71-090c6e420d4f" containerID="3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b" exitCode=0 Oct 30 22:08:54.263479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.259209 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" event={"ID":"d200a103-ab8c-45ef-ac71-090c6e420d4f","Type":"ContainerDied","Data":"3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b"} Oct 30 22:08:54.263479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.259260 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" event={"ID":"d200a103-ab8c-45ef-ac71-090c6e420d4f","Type":"ContainerDied","Data":"344f7bae5a638300289a4a7e84d57f1c741bde22e5a57c41ba77091ea53ca628"} Oct 30 22:08:54.263479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.259289 5139 scope.go:122] "RemoveContainer" containerID="3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b" Oct 30 22:08:54.263479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.259483 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq" Oct 30 22:08:54.302052 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.293051 5139 scope.go:122] "RemoveContainer" containerID="3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b" Oct 30 22:08:54.302052 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:08:54.293492 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b\": container with ID starting with 3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b not found: ID does not exist" containerID="3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b" Oct 30 22:08:54.302052 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.293535 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b"} err="failed to get container status \"3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b\": rpc error: code = NotFound desc = could not find container \"3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b\": container with ID starting with 3a9033c8ffcb14a1b6e3b674ef773722fcde8cb79c6ad5e5f3a5efe48b1b453b not found: ID does not exist" Oct 30 22:08:54.430711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.430349 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq"] Oct 30 22:08:54.473598 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:54.473564 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f9b7e7-67846f9995-r4zmq"] Oct 30 22:08:55.543922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:08:55.543886 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d200a103-ab8c-45ef-ac71-090c6e420d4f" path="/var/lib/kubelet/pods/d200a103-ab8c-45ef-ac71-090c6e420d4f/volumes" Oct 30 22:09:10.862352 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:10.862321 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb"] Oct 30 22:09:10.862928 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:10.862536 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" containerID="cri-o://a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f" gracePeriod=30 Oct 30 22:09:10.902925 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:10.902886 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh"] Oct 30 22:09:10.903500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:10.903467 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" containerID="cri-o://d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22" gracePeriod=30 Oct 30 22:09:10.904233 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:10.904211 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv"] Oct 30 22:09:10.904513 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:10.904483 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" containerID="cri-o://6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6" gracePeriod=30 Oct 30 22:09:10.908651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:10.908632 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp"] Oct 30 22:09:10.908787 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:10.908764 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" containerID="cri-o://11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b" gracePeriod=30 Oct 30 22:09:13.678042 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:13.678006 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:13.678548 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:13.678064 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:14.205703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:14.205664 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-72zzh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:14.205902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:14.205740 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:14.341560 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:14.341518 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:14.341727 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:14.341597 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:14.636374 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:14.636332 5139 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-98zzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:14.636597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:14.636438 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:16.191017 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.190980 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:09:16.258785 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.258760 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:09:16.262068 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.262044 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:09:16.265540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.265520 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:09:16.302177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302153 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d121479a-7398-4a24-bccc-283be7383d19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d121479a-7398-4a24-bccc-283be7383d19-istiod-ca-cert\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.302290 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302202 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-kube-api-access-cxsbp\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-kube-api-access-cxsbp\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.302290 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302237 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-certs\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.302290 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302269 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-token\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-token\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.302490 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302297 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-credential-socket\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.302490 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302346 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-token\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-token\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.302490 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302379 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-socket\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.302490 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302444 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-credential-socket\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.302490 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302484 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istiod-ca-cert\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.302768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302516 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-data\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.302768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302544 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-data\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.302768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302573 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istiod-ca-cert\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.302768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302605 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-envoy\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.302768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302656 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d121479a-7398-4a24-bccc-283be7383d19-istiod-ca-cert" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:16.302768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302665 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-certs\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.302768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302695 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-kube-api-access-mn7rp\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-kube-api-access-mn7rp\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.302768 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302739 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972b9d32-22c1-42f8-a368-af1b8838cf47-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972b9d32-22c1-42f8-a368-af1b8838cf47-istiod-ca-cert\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302789 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-envoy\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302818 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-envoy\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302861 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-certs\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302905 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d121479a-7398-4a24-bccc-283be7383d19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d121479a-7398-4a24-bccc-283be7383d19-istio-podinfo\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302936 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-podinfo\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302973 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-socket\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303033 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-certs\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303062 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-kube-api-access-mqltv\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-kube-api-access-mqltv\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303110 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-istio-token\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-istio-token\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.303163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303157 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-credential-socket\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303199 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-kube-api-access-h9txn\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-kube-api-access-h9txn\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303236 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-socket\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303269 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-token\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-token\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303309 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-podinfo\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303347 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-podinfo\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303406 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-data\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303457 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-credential-socket\") pod \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\" (UID: \"53278e91-ddd0-4dbb-920e-12c2bc1b7658\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303538 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-envoy\") pod \"d121479a-7398-4a24-bccc-283be7383d19\" (UID: \"d121479a-7398-4a24-bccc-283be7383d19\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303588 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-data\") pod \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\" (UID: \"f3a79712-1ddf-4042-8bd3-922bc0e399c6\") " Oct 30 22:09:16.303700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303617 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-socket\") pod \"972b9d32-22c1-42f8-a368-af1b8838cf47\" (UID: \"972b9d32-22c1-42f8-a368-af1b8838cf47\") " Oct 30 22:09:16.304470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.304448 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d121479a-7398-4a24-bccc-283be7383d19-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.307780 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302651 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-credential-socket" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302693 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-socket" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302843 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-credential-socket" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.302864 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-data" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307825 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-certs" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307839 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-socket" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303059 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-certs" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303061 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-data" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303120 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istiod-ca-cert" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303374 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istiod-ca-cert" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303637 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-credential-socket" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.307890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303808 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-socket" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.303968 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-certs" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.304736 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-socket" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.304894 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-token" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.305052 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-kube-api-access-cxsbp" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "kube-api-access-cxsbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.305131 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-token" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.305340 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-credential-socket" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.305507 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-data" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307948 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d121479a-7398-4a24-bccc-283be7383d19-istio-podinfo" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.305569 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-envoy" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.305627 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-podinfo" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.305821 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-certs" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.306378 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-kube-api-access-mqltv" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "kube-api-access-mqltv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.306677 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-envoy" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307053 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972b9d32-22c1-42f8-a368-af1b8838cf47-istiod-ca-cert" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307154 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-kube-api-access-h9txn" pod "53278e91-ddd0-4dbb-920e-12c2bc1b7658" (UID: "53278e91-ddd0-4dbb-920e-12c2bc1b7658"). InnerVolumeSpecName "kube-api-access-h9txn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307399 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-data" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307593 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-istio-token" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307610 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-podinfo" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:16.308316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.307719 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-token" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.308979 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.308457 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-envoy" pod "d121479a-7398-4a24-bccc-283be7383d19" (UID: "d121479a-7398-4a24-bccc-283be7383d19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.308979 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.308652 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-kube-api-access-mn7rp" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "kube-api-access-mn7rp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:16.309129 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.309112 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-envoy" pod "972b9d32-22c1-42f8-a368-af1b8838cf47" (UID: "972b9d32-22c1-42f8-a368-af1b8838cf47"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:16.309206 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.309189 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-podinfo" pod "f3a79712-1ddf-4042-8bd3-922bc0e399c6" (UID: "f3a79712-1ddf-4042-8bd3-922bc0e399c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.422671 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.422707 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.422725 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.422993 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.423105 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.423120 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxsbp\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-kube-api-access-cxsbp\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427342 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427360 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427376 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427391 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427405 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427435 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427449 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427464 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427477 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427493 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427512 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427526 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427540 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn7rp\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-kube-api-access-mn7rp\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427556 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972b9d32-22c1-42f8-a368-af1b8838cf47-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427570 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427585 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427599 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972b9d32-22c1-42f8-a368-af1b8838cf47-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427612 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d121479a-7398-4a24-bccc-283be7383d19-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427625 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53278e91-ddd0-4dbb-920e-12c2bc1b7658-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427640 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53278e91-ddd0-4dbb-920e-12c2bc1b7658-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427654 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427668 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqltv\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-kube-api-access-mqltv\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427682 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d121479a-7398-4a24-bccc-283be7383d19-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427696 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d121479a-7398-4a24-bccc-283be7383d19-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427711 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9txn\" (UniqueName: \"kubernetes.io/projected/53278e91-ddd0-4dbb-920e-12c2bc1b7658-kube-api-access-h9txn\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427725 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3a79712-1ddf-4042-8bd3-922bc0e399c6-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427739 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427753 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3a79712-1ddf-4042-8bd3-922bc0e399c6-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.427846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.427766 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972b9d32-22c1-42f8-a368-af1b8838cf47-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:16.469555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.469498 5139 generic.go:356] "Generic (PLEG): container finished" podID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerID="11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b" exitCode=0 Oct 30 22:09:16.469753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.469636 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" event={"ID":"53278e91-ddd0-4dbb-920e-12c2bc1b7658","Type":"ContainerDied","Data":"11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b"} Oct 30 22:09:16.469753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.469714 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" event={"ID":"53278e91-ddd0-4dbb-920e-12c2bc1b7658","Type":"ContainerDied","Data":"af09c420067da56af92d27a69567c4898f6fdfd81208cc4d5a2d0744f227e008"} Oct 30 22:09:16.469927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.469749 5139 scope.go:122] "RemoveContainer" containerID="11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b" Oct 30 22:09:16.469983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.469932 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp" Oct 30 22:09:16.475693 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.475663 5139 generic.go:356] "Generic (PLEG): container finished" podID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerID="6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6" exitCode=0 Oct 30 22:09:16.475859 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.475842 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" event={"ID":"f3a79712-1ddf-4042-8bd3-922bc0e399c6","Type":"ContainerDied","Data":"6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6"} Oct 30 22:09:16.475924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.475883 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" event={"ID":"f3a79712-1ddf-4042-8bd3-922bc0e399c6","Type":"ContainerDied","Data":"4b86b499ec71cd0e6676255e158d11be931a76b315312e8ae3ecf9a11d386f7c"} Oct 30 22:09:16.475977 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.475968 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv" Oct 30 22:09:16.481278 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.481255 5139 generic.go:356] "Generic (PLEG): container finished" podID="d121479a-7398-4a24-bccc-283be7383d19" containerID="d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22" exitCode=0 Oct 30 22:09:16.481449 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.481357 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" event={"ID":"d121479a-7398-4a24-bccc-283be7383d19","Type":"ContainerDied","Data":"d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22"} Oct 30 22:09:16.481449 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.481404 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" event={"ID":"d121479a-7398-4a24-bccc-283be7383d19","Type":"ContainerDied","Data":"ed52b953cbb9d76f378496b21aef4fbc4f3deee669fe5f77d0c4655c9fea3537"} Oct 30 22:09:16.481595 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.481505 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh" Oct 30 22:09:16.487997 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.487972 5139 generic.go:356] "Generic (PLEG): container finished" podID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerID="a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f" exitCode=0 Oct 30 22:09:16.488130 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.488029 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" event={"ID":"972b9d32-22c1-42f8-a368-af1b8838cf47","Type":"ContainerDied","Data":"a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f"} Oct 30 22:09:16.488130 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.488065 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" event={"ID":"972b9d32-22c1-42f8-a368-af1b8838cf47","Type":"ContainerDied","Data":"447fae7191c4740dd4dfb91e568eae0f35f80400d08b668b9e24d0b876cd2926"} Oct 30 22:09:16.488130 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.488123 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb" Oct 30 22:09:16.493625 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.493550 5139 scope.go:122] "RemoveContainer" containerID="11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b" Oct 30 22:09:16.494841 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:16.494805 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b\": container with ID starting with 11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b not found: ID does not exist" containerID="11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b" Oct 30 22:09:16.494971 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.494838 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b"} err="failed to get container status \"11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b\": rpc error: code = NotFound desc = could not find container \"11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b\": container with ID starting with 11a4e2a5fe317565cc825292c94a6384bead07b31e29de4770dbfa908585de9b not found: ID does not exist" Oct 30 22:09:16.494971 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.494862 5139 scope.go:122] "RemoveContainer" containerID="6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6" Oct 30 22:09:16.515262 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.515231 5139 scope.go:122] "RemoveContainer" containerID="6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6" Oct 30 22:09:16.515834 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:16.515677 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6\": container with ID starting with 6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6 not found: ID does not exist" containerID="6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6" Oct 30 22:09:16.515834 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.515718 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6"} err="failed to get container status \"6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6\": rpc error: code = NotFound desc = could not find container \"6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6\": container with ID starting with 6306ed11e6a6908e8731f238a59eb4156250a32f8e66f182d8db91b3cebbc5e6 not found: ID does not exist" Oct 30 22:09:16.515834 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.515743 5139 scope.go:122] "RemoveContainer" containerID="d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22" Oct 30 22:09:16.526575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.526553 5139 scope.go:122] "RemoveContainer" containerID="d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22" Oct 30 22:09:16.526932 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:16.526900 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22\": container with ID starting with d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22 not found: ID does not exist" containerID="d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22" Oct 30 22:09:16.527027 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.526940 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22"} err="failed to get container status \"d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22\": rpc error: code = NotFound desc = could not find container \"d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22\": container with ID starting with d6f39776836c17bc76c22d710c91df47073c0121b05f021e6981ea5219a5ec22 not found: ID does not exist" Oct 30 22:09:16.527027 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.526967 5139 scope.go:122] "RemoveContainer" containerID="a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f" Oct 30 22:09:16.553380 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.553353 5139 scope.go:122] "RemoveContainer" containerID="a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f" Oct 30 22:09:16.553788 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:16.553759 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f\": container with ID starting with a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f not found: ID does not exist" containerID="a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f" Oct 30 22:09:16.553892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.553800 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f"} err="failed to get container status \"a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f\": rpc error: code = NotFound desc = could not find container \"a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f\": container with ID starting with a8e00845be575e3754627be1d88233119b5048a38ef3f0c3b53df3d5f0aec73f not found: ID does not exist" Oct 30 22:09:16.904086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.904042 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb"] Oct 30 22:09:16.907696 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:16.907672 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-pnsfb"] Oct 30 22:09:17.011642 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.011607 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv"] Oct 30 22:09:17.016323 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.016296 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-98zzv"] Oct 30 22:09:17.133538 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.133502 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh"] Oct 30 22:09:17.142822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.142745 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-72zzh"] Oct 30 22:09:17.248561 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.248521 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp"] Oct 30 22:09:17.252359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.252328 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lgjkp"] Oct 30 22:09:17.543866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.543822 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" path="/var/lib/kubelet/pods/53278e91-ddd0-4dbb-920e-12c2bc1b7658/volumes" Oct 30 22:09:17.544450 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.544408 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" path="/var/lib/kubelet/pods/972b9d32-22c1-42f8-a368-af1b8838cf47/volumes" Oct 30 22:09:17.544911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.544895 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d121479a-7398-4a24-bccc-283be7383d19" path="/var/lib/kubelet/pods/d121479a-7398-4a24-bccc-283be7383d19/volumes" Oct 30 22:09:17.545675 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:17.545659 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" path="/var/lib/kubelet/pods/f3a79712-1ddf-4042-8bd3-922bc0e399c6/volumes" Oct 30 22:09:19.805514 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:19.805471 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:09:28.246472 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:28.246437 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp"] Oct 30 22:09:28.246902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:28.246718 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" containerID="cri-o://7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968" gracePeriod=30 Oct 30 22:09:29.111463 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:29.111405 5139 patch_prober.go:29] interesting pod/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:09:29.111631 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:29.111496 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:09:30.051525 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:30.051485 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj"] Oct 30 22:09:30.051921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:30.051709 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" containerID="cri-o://f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d" gracePeriod=30 Oct 30 22:09:30.120723 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:30.098391 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr"] Oct 30 22:09:30.120723 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:30.098701 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" containerID="cri-o://65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7" gracePeriod=30 Oct 30 22:09:31.145581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.145544 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq"] Oct 30 22:09:31.146012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.145744 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerName="maas-api" containerID="cri-o://05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910" gracePeriod=30 Oct 30 22:09:31.426442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.426403 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:09:31.539590 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.539554 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-metrics-certs\") pod \"c5cd2838-6854-4690-97ff-06e1bb784f68\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " Oct 30 22:09:31.539873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.539647 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5cd2838-6854-4690-97ff-06e1bb784f68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cd2838-6854-4690-97ff-06e1bb784f68-openshift-service-ca\") pod \"c5cd2838-6854-4690-97ff-06e1bb784f68\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " Oct 30 22:09:31.539873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.539838 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls\") pod \"c5cd2838-6854-4690-97ff-06e1bb784f68\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " Oct 30 22:09:31.540074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.539997 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5cd2838-6854-4690-97ff-06e1bb784f68-kube-api-access-7dhlm\" (UniqueName: \"kubernetes.io/projected/c5cd2838-6854-4690-97ff-06e1bb784f68-kube-api-access-7dhlm\") pod \"c5cd2838-6854-4690-97ff-06e1bb784f68\" (UID: \"c5cd2838-6854-4690-97ff-06e1bb784f68\") " Oct 30 22:09:31.541699 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.540565 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5cd2838-6854-4690-97ff-06e1bb784f68-openshift-service-ca" pod "c5cd2838-6854-4690-97ff-06e1bb784f68" (UID: "c5cd2838-6854-4690-97ff-06e1bb784f68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:31.543952 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.543918 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5cd2838-6854-4690-97ff-06e1bb784f68-kube-api-access-7dhlm" pod "c5cd2838-6854-4690-97ff-06e1bb784f68" (UID: "c5cd2838-6854-4690-97ff-06e1bb784f68"). InnerVolumeSpecName "kube-api-access-7dhlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:31.545193 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.545165 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls" pod "c5cd2838-6854-4690-97ff-06e1bb784f68" (UID: "c5cd2838-6854-4690-97ff-06e1bb784f68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:31.546779 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.545477 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-metrics-certs" pod "c5cd2838-6854-4690-97ff-06e1bb784f68" (UID: "c5cd2838-6854-4690-97ff-06e1bb784f68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:09:31.594742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.594699 5139 generic.go:356] "Generic (PLEG): container finished" podID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerID="05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910" exitCode=0 Oct 30 22:09:31.594931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.594794 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" event={"ID":"c5cd2838-6854-4690-97ff-06e1bb784f68","Type":"ContainerDied","Data":"05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910"} Oct 30 22:09:31.594931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.594849 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" event={"ID":"c5cd2838-6854-4690-97ff-06e1bb784f68","Type":"ContainerDied","Data":"7e0fabe1858b92ed9e007baad991dcf2de52a72773302b33f66e9cbdbc2947bc"} Oct 30 22:09:31.594931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.594882 5139 scope.go:122] "RemoveContainer" containerID="05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910" Oct 30 22:09:31.595534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.595507 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq" Oct 30 22:09:31.610938 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.610917 5139 scope.go:122] "RemoveContainer" containerID="db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa" Oct 30 22:09:31.622493 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.622461 5139 scope.go:122] "RemoveContainer" containerID="05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910" Oct 30 22:09:31.623404 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:31.623381 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910\": container with ID starting with 05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910 not found: ID does not exist" containerID="05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910" Oct 30 22:09:31.623525 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.623448 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910"} err="failed to get container status \"05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910\": rpc error: code = NotFound desc = could not find container \"05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910\": container with ID starting with 05a71ba3415a93db142af2a3cd1a48514fd5a0d109fc22f5ddb58d41f8fcf910 not found: ID does not exist" Oct 30 22:09:31.623525 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.623469 5139 scope.go:122] "RemoveContainer" containerID="db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa" Oct 30 22:09:31.623820 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:31.623804 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa\": container with ID starting with db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa not found: ID does not exist" containerID="db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa" Oct 30 22:09:31.623896 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.623822 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa"} err="failed to get container status \"db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa\": rpc error: code = NotFound desc = could not find container \"db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa\": container with ID starting with db38e99de70be1bc4252e9901bea75e34a56ed62fb9f5b91d470de8defd07afa not found: ID does not exist" Oct 30 22:09:31.642296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.642263 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:31.642296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.642294 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhlm\" (UniqueName: \"kubernetes.io/projected/c5cd2838-6854-4690-97ff-06e1bb784f68-kube-api-access-7dhlm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:31.642296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.642303 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c5cd2838-6854-4690-97ff-06e1bb784f68-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:31.642523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.642312 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5cd2838-6854-4690-97ff-06e1bb784f68-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:31.760258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.760225 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq"] Oct 30 22:09:31.773492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:31.773464 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-82b62e-68447f9c45-8qqdq"] Oct 30 22:09:33.532125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.532095 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:09:33.543597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.543569 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" path="/var/lib/kubelet/pods/c5cd2838-6854-4690-97ff-06e1bb784f68/volumes" Oct 30 22:09:33.608464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.608431 5139 generic.go:356] "Generic (PLEG): container finished" podID="8dd019f2-888d-483b-8d66-51f93f350648" containerID="7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968" exitCode=0 Oct 30 22:09:33.608647 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.608483 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" Oct 30 22:09:33.608647 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.608506 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" event={"ID":"8dd019f2-888d-483b-8d66-51f93f350648","Type":"ContainerDied","Data":"7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968"} Oct 30 22:09:33.608647 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.608570 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp" event={"ID":"8dd019f2-888d-483b-8d66-51f93f350648","Type":"ContainerDied","Data":"a1c5f9173b5eb4a065cb3d3dea9bb421af74ba6f64d3c95c100f8eb7d70aa9af"} Oct 30 22:09:33.608647 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.608601 5139 scope.go:122] "RemoveContainer" containerID="7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968" Oct 30 22:09:33.618063 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.618043 5139 scope.go:122] "RemoveContainer" containerID="7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968" Oct 30 22:09:33.618348 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:33.618328 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968\": container with ID starting with 7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968 not found: ID does not exist" containerID="7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968" Oct 30 22:09:33.618457 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.618360 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968"} err="failed to get container status \"7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968\": rpc error: code = NotFound desc = could not find container \"7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968\": container with ID starting with 7b26083d5fca222adde96c7f4c508c72403f71b2996b1d4adb1b633c5d6ab968 not found: ID does not exist" Oct 30 22:09:33.668826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.668795 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-envoy\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.668999 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.668847 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-data\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.668999 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.668944 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-socket\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.669117 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.668999 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-istio-token\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-istio-token\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.669117 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669077 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dd019f2-888d-483b-8d66-51f93f350648-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd019f2-888d-483b-8d66-51f93f350648-istio-podinfo\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.669225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669125 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-credential-socket\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.669225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669159 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd019f2-888d-483b-8d66-51f93f350648-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd019f2-888d-483b-8d66-51f93f350648-istiod-ca-cert\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.669225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669186 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-data" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:33.669366 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669194 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-certs\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.669366 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669247 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-socket" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:33.669366 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669278 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-kube-api-access-r5w5c\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-kube-api-access-r5w5c\") pod \"8dd019f2-888d-483b-8d66-51f93f350648\" (UID: \"8dd019f2-888d-483b-8d66-51f93f350648\") " Oct 30 22:09:33.669564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669429 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-credential-socket" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:33.669564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669449 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-certs" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:33.669666 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.669627 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd019f2-888d-483b-8d66-51f93f350648-istiod-ca-cert" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:33.670238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.670214 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:33.670238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.670239 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dd019f2-888d-483b-8d66-51f93f350648-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:33.670382 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.670253 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:33.670382 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.670266 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:33.670382 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.670280 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:33.671648 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.671623 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-envoy" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:33.671910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.671881 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dd019f2-888d-483b-8d66-51f93f350648-istio-podinfo" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:33.671910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.671901 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-kube-api-access-r5w5c" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "kube-api-access-r5w5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:33.672040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.671942 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-istio-token" pod "8dd019f2-888d-483b-8d66-51f93f350648" (UID: "8dd019f2-888d-483b-8d66-51f93f350648"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:33.771960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.771870 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dd019f2-888d-483b-8d66-51f93f350648-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:33.771960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.771900 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:33.771960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.771912 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dd019f2-888d-483b-8d66-51f93f350648-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:33.771960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:33.771924 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5w5c\" (UniqueName: \"kubernetes.io/projected/8dd019f2-888d-483b-8d66-51f93f350648-kube-api-access-r5w5c\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:34.023530 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:34.023435 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp"] Oct 30 22:09:34.027478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:34.027448 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-a5f7b9-openshift-default-99f7c85d7-c66kp"] Oct 30 22:09:35.372476 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.372449 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 22:09:35.414299 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.414273 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 22:09:35.492643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492594 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istiod-ca-cert\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.492826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492657 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-podinfo\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.492826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492683 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b2e91287-9971-475c-a10f-c95f3f346b59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e91287-9971-475c-a10f-c95f3f346b59-istio-podinfo\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.492826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492737 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-credential-socket\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.492826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492766 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-envoy\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.492826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492812 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-data\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492837 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-envoy\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492890 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-data\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492931 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-istio-token\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-istio-token\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492968 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istiod-ca-cert" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.492999 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-credential-socket\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493071 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-token\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-token\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493117 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-certs\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493149 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-kube-api-access-r7z9p\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-kube-api-access-r7z9p\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493187 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-socket\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493188 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-credential-socket" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493246 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-socket\") pod \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\" (UID: \"cfe38e7c-911f-432d-a9b6-90bc9b31de5c\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493270 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-kube-api-access-kfk5c\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-kube-api-access-kfk5c\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493331 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-data" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493338 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-certs\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493407 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-credential-socket" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493451 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e91287-9971-475c-a10f-c95f3f346b59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e91287-9971-475c-a10f-c95f3f346b59-istiod-ca-cert\") pod \"b2e91287-9971-475c-a10f-c95f3f346b59\" (UID: \"b2e91287-9971-475c-a10f-c95f3f346b59\") " Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493568 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-certs" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493615 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-certs" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493669 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-data" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.493882 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-socket" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494209 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494231 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494245 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494259 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494273 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494286 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494301 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494316 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494395 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e91287-9971-475c-a10f-c95f3f346b59-istiod-ca-cert" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.494535 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-socket" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.496087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.496033 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-envoy" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.497575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.496505 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-token" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:35.497575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.496719 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-kube-api-access-r7z9p" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "kube-api-access-r7z9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:35.497575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.496816 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-envoy" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:09:35.497575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.496892 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-podinfo" pod "cfe38e7c-911f-432d-a9b6-90bc9b31de5c" (UID: "cfe38e7c-911f-432d-a9b6-90bc9b31de5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:35.497575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.496988 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b2e91287-9971-475c-a10f-c95f3f346b59-istio-podinfo" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:09:35.497575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.497426 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-istio-token" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:35.498028 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.498014 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-kube-api-access-kfk5c" pod "b2e91287-9971-475c-a10f-c95f3f346b59" (UID: "b2e91287-9971-475c-a10f-c95f3f346b59"). InnerVolumeSpecName "kube-api-access-kfk5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:09:35.545876 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.545822 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd019f2-888d-483b-8d66-51f93f350648" path="/var/lib/kubelet/pods/8dd019f2-888d-483b-8d66-51f93f350648/volumes" Oct 30 22:09:35.595998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.595920 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.595998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.595959 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7z9p\" (UniqueName: \"kubernetes.io/projected/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-kube-api-access-r7z9p\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.595998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.595975 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.595998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.595989 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfk5c\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-kube-api-access-kfk5c\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.596230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.596002 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b2e91287-9971-475c-a10f-c95f3f346b59-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.596230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.596016 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.596230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.596028 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b2e91287-9971-475c-a10f-c95f3f346b59-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.596230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.596042 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe38e7c-911f-432d-a9b6-90bc9b31de5c-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.596230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.596057 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b2e91287-9971-475c-a10f-c95f3f346b59-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.596230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.596071 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b2e91287-9971-475c-a10f-c95f3f346b59-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:09:35.622913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.622880 5139 generic.go:356] "Generic (PLEG): container finished" podID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerID="f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d" exitCode=0 Oct 30 22:09:35.623081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.622932 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" Oct 30 22:09:35.623081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.622968 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" event={"ID":"cfe38e7c-911f-432d-a9b6-90bc9b31de5c","Type":"ContainerDied","Data":"f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d"} Oct 30 22:09:35.623081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.623040 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj" event={"ID":"cfe38e7c-911f-432d-a9b6-90bc9b31de5c","Type":"ContainerDied","Data":"7c5653c1ee7a5ce36486766becb9fb327abade2af1371d53fa329bc8f899eea2"} Oct 30 22:09:35.623081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.623074 5139 scope.go:122] "RemoveContainer" containerID="f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d" Oct 30 22:09:35.624383 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.624357 5139 generic.go:356] "Generic (PLEG): container finished" podID="b2e91287-9971-475c-a10f-c95f3f346b59" containerID="65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7" exitCode=0 Oct 30 22:09:35.624520 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.624399 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" Oct 30 22:09:35.624520 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.624453 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" event={"ID":"b2e91287-9971-475c-a10f-c95f3f346b59","Type":"ContainerDied","Data":"65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7"} Oct 30 22:09:35.624520 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.624500 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr" event={"ID":"b2e91287-9971-475c-a10f-c95f3f346b59","Type":"ContainerDied","Data":"9456f35653e64344823023002185ecf66b4cc8dc2e87eff4d0be63b3c3b88a0e"} Oct 30 22:09:35.632640 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.632624 5139 scope.go:122] "RemoveContainer" containerID="f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d" Oct 30 22:09:35.632923 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:35.632897 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d\": container with ID starting with f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d not found: ID does not exist" containerID="f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d" Oct 30 22:09:35.632959 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.632931 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d"} err="failed to get container status \"f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d\": rpc error: code = NotFound desc = could not find container \"f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d\": container with ID starting with f71ea50af2f1c47b6252db8d0aa7c16c5551d8dc88fed446cc49a76786476f9d not found: ID does not exist" Oct 30 22:09:35.632959 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.632947 5139 scope.go:122] "RemoveContainer" containerID="65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7" Oct 30 22:09:35.641728 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.641711 5139 scope.go:122] "RemoveContainer" containerID="65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7" Oct 30 22:09:35.642002 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:35.641978 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7\": container with ID starting with 65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7 not found: ID does not exist" containerID="65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7" Oct 30 22:09:35.642099 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.642004 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7"} err="failed to get container status \"65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7\": rpc error: code = NotFound desc = could not find container \"65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7\": container with ID starting with 65102a2ad1ed4d4dc07956ec5f7763d36b4db6c6ca68c1c70557796a97f5a2f7 not found: ID does not exist" Oct 30 22:09:35.766375 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.757613 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj"] Oct 30 22:09:35.767984 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.767161 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-wflhj"] Oct 30 22:09:35.864934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.864842 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr"] Oct 30 22:09:35.869879 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:35.869845 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-hcscr"] Oct 30 22:09:37.544039 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:37.544007 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" path="/var/lib/kubelet/pods/b2e91287-9971-475c-a10f-c95f3f346b59/volumes" Oct 30 22:09:37.544484 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:09:37.544469 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" path="/var/lib/kubelet/pods/cfe38e7c-911f-432d-a9b6-90bc9b31de5c/volumes" Oct 30 22:09:49.804154 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:09:49.804112 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:10:19.807926 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:10:19.807878 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:10:29.472500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:29.472459 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw"] Oct 30 22:10:29.473387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:29.472674 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" containerID="cri-o://665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d" gracePeriod=30 Oct 30 22:10:34.609182 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.609143 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 22:10:34.609701 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.609218 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 22:10:34.755802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.755774 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:10:34.899545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.899497 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-socket\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.899545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.899550 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21d25c30-87c6-4eff-857c-87973d3bbfab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d25c30-87c6-4eff-857c-87973d3bbfab-istiod-ca-cert\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.899810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.899647 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-podinfo\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.899810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.899749 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-token\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-token\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.899924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.899843 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-socket" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:34.899924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.899872 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-certs\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.900027 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.899973 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-credential-socket\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.900077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900040 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-data\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.900128 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900079 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-certs" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:34.900128 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900109 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-envoy\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.900238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900154 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-kube-api-access-bjdkk\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-kube-api-access-bjdkk\") pod \"21d25c30-87c6-4eff-857c-87973d3bbfab\" (UID: \"21d25c30-87c6-4eff-857c-87973d3bbfab\") " Oct 30 22:10:34.900238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900105 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21d25c30-87c6-4eff-857c-87973d3bbfab-istiod-ca-cert" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:10:34.900238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900219 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-credential-socket" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:34.900368 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900302 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-data" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:34.900722 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900700 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:34.900871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900730 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:34.900871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900746 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:34.900871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900761 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21d25c30-87c6-4eff-857c-87973d3bbfab-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:34.900871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.900782 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:34.901976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.901949 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-token" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:34.902397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.902369 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-kube-api-access-bjdkk" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "kube-api-access-bjdkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:10:34.902517 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.902464 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-envoy" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:10:34.902581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:34.902539 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-podinfo" pod "21d25c30-87c6-4eff-857c-87973d3bbfab" (UID: "21d25c30-87c6-4eff-857c-87973d3bbfab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:10:35.002570 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.002537 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:35.002570 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.002567 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjdkk\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-kube-api-access-bjdkk\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:35.002570 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.002576 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:35.002812 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.002585 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21d25c30-87c6-4eff-857c-87973d3bbfab-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:10:35.022081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.022046 5139 generic.go:356] "Generic (PLEG): container finished" podID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerID="665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d" exitCode=0 Oct 30 22:10:35.022263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.022102 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" Oct 30 22:10:35.022263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.022129 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" event={"ID":"21d25c30-87c6-4eff-857c-87973d3bbfab","Type":"ContainerDied","Data":"665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d"} Oct 30 22:10:35.022263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.022205 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw" event={"ID":"21d25c30-87c6-4eff-857c-87973d3bbfab","Type":"ContainerDied","Data":"b3fe8ed6805235c55caffd928dbd59cc17dc1910f2720f65c66f2540e8738d01"} Oct 30 22:10:35.022263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.022233 5139 scope.go:122] "RemoveContainer" containerID="665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d" Oct 30 22:10:35.033055 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.033032 5139 scope.go:122] "RemoveContainer" containerID="665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d" Oct 30 22:10:35.033370 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:10:35.033354 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d\": container with ID starting with 665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d not found: ID does not exist" containerID="665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d" Oct 30 22:10:35.033434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.033380 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d"} err="failed to get container status \"665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d\": rpc error: code = NotFound desc = could not find container \"665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d\": container with ID starting with 665a0ac466df7a91ea69eb8626a1a8b4865494a6fe2cd647ea40f74cc89e056d not found: ID does not exist" Oct 30 22:10:35.186440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.186385 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw"] Oct 30 22:10:35.191564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.191534 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5g7qw"] Oct 30 22:10:35.542687 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:10:35.542654 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" path="/var/lib/kubelet/pods/21d25c30-87c6-4eff-857c-87973d3bbfab/volumes" Oct 30 22:10:49.805015 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:10:49.804972 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:11:19.804044 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:11:19.804005 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:11:27.882458 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882423 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt"] Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882913 5139 cpu_manager.go:414] "containerMap: removing container" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882928 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882939 5139 cpu_manager.go:414] "containerMap: removing container" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882944 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882952 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882957 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882965 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882971 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882978 5139 cpu_manager.go:414] "containerMap: removing container" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882982 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.882993 5139 cpu_manager.go:414] "containerMap: removing container" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883000 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883007 5139 cpu_manager.go:414] "containerMap: removing container" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883012 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883020 5139 cpu_manager.go:414] "containerMap: removing container" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883024 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883032 5139 cpu_manager.go:414] "containerMap: removing container" podUID="d200a103-ab8c-45ef-ac71-090c6e420d4f" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883036 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d200a103-ab8c-45ef-ac71-090c6e420d4f" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883044 5139 cpu_manager.go:414] "containerMap: removing container" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883048 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883054 5139 cpu_manager.go:414] "containerMap: removing container" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883058 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883073 5139 cpu_manager.go:414] "containerMap: removing container" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883077 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883091 5139 cpu_manager.go:414] "containerMap: removing container" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883096 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883102 5139 cpu_manager.go:414] "containerMap: removing container" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883107 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883115 5139 cpu_manager.go:414] "containerMap: removing container" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883119 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883207 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94b7c7c-9e17-4087-b287-ffe225700146" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883214 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="21d25c30-87c6-4eff-857c-87973d3bbfab" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883221 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a79712-1ddf-4042-8bd3-922bc0e399c6" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883227 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="d121479a-7398-4a24-bccc-283be7383d19" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883235 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="27903e11-7864-4b21-84a8-17ed8c9ca170" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883240 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="231eef6e-a1f5-44fa-a50f-ffcff69259dd" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883245 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883251 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5cd2838-6854-4690-97ff-06e1bb784f68" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883256 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="972b9d32-22c1-42f8-a368-af1b8838cf47" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883261 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe38e7c-911f-432d-a9b6-90bc9b31de5c" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883265 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="53278e91-ddd0-4dbb-920e-12c2bc1b7658" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883270 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf198bb7-20b5-43d9-a2ba-f17d2b19d40c" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883275 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="d200a103-ab8c-45ef-ac71-090c6e420d4f" containerName="maas-api" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883281 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e91287-9971-475c-a10f-c95f3f346b59" containerName="istio-proxy" Oct 30 22:11:27.884810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.883286 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd019f2-888d-483b-8d66-51f93f350648" containerName="istio-proxy" Oct 30 22:11:27.886387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.886372 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:27.929899 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.929868 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7w8h\" (UniqueName: \"kubernetes.io/projected/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-kube-api-access-l7w8h\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:27.930052 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.930022 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-metrics-certs\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:27.930109 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.930059 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:27.930109 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.930092 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:27.935579 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.935558 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-a3aa65-serving-cert\"" Oct 30 22:11:27.984605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:27.984573 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt"] Oct 30 22:11:28.030951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.030922 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7w8h\" (UniqueName: \"kubernetes.io/projected/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-kube-api-access-l7w8h\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.031120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.031027 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-metrics-certs\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.031120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.031063 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.031120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.031101 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.031299 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:11:28.031283 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-serving-cert: secret "maas-api-e2e-api-b-a3aa65-serving-cert" not found Oct 30 22:11:28.031380 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:11:28.031366 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls podName:a4082918-3ac0-4ca5-aa66-2f3dee8969ab nodeName:}" failed. No retries permitted until 2026-10-30 22:11:28.531345487 +0000 UTC m=+1997.122630971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls") pod "maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" (UID: "a4082918-3ac0-4ca5-aa66-2f3dee8969ab") : secret "maas-api-e2e-api-b-a3aa65-serving-cert" not found Oct 30 22:11:28.032176 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.032150 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-openshift-service-ca\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.033835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.033813 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-metrics-certs\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.098986 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.098950 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7w8h\" (UniqueName: \"kubernetes.io/projected/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-kube-api-access-l7w8h\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.538056 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.538020 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.540499 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.540466 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls\") pod \"maas-api-e2e-api-b-a3aa65-67849d9545-5njxt\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:28.799992 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:28.799799 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:29.194059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:29.193978 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt"] Oct 30 22:11:29.199961 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:11:29.199934 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4082918_3ac0_4ca5_aa66_2f3dee8969ab.slice/crio-2094cb6cb164fce4f9cd74b4bd0d96b9cf37ebe6b1256c4f5a381cfe71f3a7f8 WatchSource:0}: Error finding container 2094cb6cb164fce4f9cd74b4bd0d96b9cf37ebe6b1256c4f5a381cfe71f3a7f8: Status 404 returned error can't find the container with id 2094cb6cb164fce4f9cd74b4bd0d96b9cf37ebe6b1256c4f5a381cfe71f3a7f8 Oct 30 22:11:29.201693 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:29.201672 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:11:29.321798 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:29.321759 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" event={"ID":"a4082918-3ac0-4ca5-aa66-2f3dee8969ab","Type":"ContainerStarted","Data":"2094cb6cb164fce4f9cd74b4bd0d96b9cf37ebe6b1256c4f5a381cfe71f3a7f8"} Oct 30 22:11:30.328530 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:30.328502 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" event={"ID":"a4082918-3ac0-4ca5-aa66-2f3dee8969ab","Type":"ContainerStarted","Data":"a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9"} Oct 30 22:11:30.328852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:30.328615 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:30.478786 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:30.478733 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" podStartSLOduration=2.417447032 podStartE2EDuration="3.478719963s" totalImagesPullingTime="1.061272931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:30.478137608 +0000 UTC m=+1999.069423096" watchObservedRunningTime="2026-10-30 22:11:30.478719963 +0000 UTC m=+1999.070005449" Oct 30 22:11:36.339347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:11:36.339315 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:11:49.805523 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:11:49.805483 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:12:19.804679 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:12:19.804639 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:12:26.348814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.348764 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb"] Oct 30 22:12:26.353138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.353115 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.402128 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.402088 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-95b84c-serving-cert\"" Oct 30 22:12:26.423813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.423778 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.423813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.423814 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.424058 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.423838 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72p2r\" (UniqueName: \"kubernetes.io/projected/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-kube-api-access-72p2r\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.424058 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.423996 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-metrics-certs\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.451022 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.450986 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb"] Oct 30 22:12:26.524843 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.524811 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-metrics-certs\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.525035 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.524916 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.525035 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.524947 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.525035 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.524966 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72p2r\" (UniqueName: \"kubernetes.io/projected/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-kube-api-access-72p2r\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.525233 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:12:26.525077 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-serving-cert: secret "maas-api-e2e-ipp-a-95b84c-serving-cert" not found Oct 30 22:12:26.525233 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:12:26.525153 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls podName:ac3b2a5a-8e7d-4131-9354-4aac1f436c08 nodeName:}" failed. No retries permitted until 2026-10-30 22:12:27.025133956 +0000 UTC m=+2055.616419424 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls") pod "maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" (UID: "ac3b2a5a-8e7d-4131-9354-4aac1f436c08") : secret "maas-api-e2e-ipp-a-95b84c-serving-cert" not found Oct 30 22:12:26.525752 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.525726 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.527453 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.527431 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-metrics-certs\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:26.590716 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:26.590678 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72p2r\" (UniqueName: \"kubernetes.io/projected/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-kube-api-access-72p2r\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:27.032184 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:27.032145 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:27.034689 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:27.034664 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls\") pod \"maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:27.266611 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:27.266570 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:27.640269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:27.640106 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb"] Oct 30 22:12:27.645182 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:12:27.645154 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3b2a5a_8e7d_4131_9354_4aac1f436c08.slice/crio-c9f1fdf73916f3a78ac4712aac43b7bc41bc0c9b13fc45cff877af29bcc85c16 WatchSource:0}: Error finding container c9f1fdf73916f3a78ac4712aac43b7bc41bc0c9b13fc45cff877af29bcc85c16: Status 404 returned error can't find the container with id c9f1fdf73916f3a78ac4712aac43b7bc41bc0c9b13fc45cff877af29bcc85c16 Oct 30 22:12:28.647734 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:28.647695 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" event={"ID":"ac3b2a5a-8e7d-4131-9354-4aac1f436c08","Type":"ContainerStarted","Data":"476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9"} Oct 30 22:12:28.648105 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:28.647759 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" event={"ID":"ac3b2a5a-8e7d-4131-9354-4aac1f436c08","Type":"ContainerStarted","Data":"c9f1fdf73916f3a78ac4712aac43b7bc41bc0c9b13fc45cff877af29bcc85c16"} Oct 30 22:12:28.648105 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:28.647791 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:28.784461 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:28.784375 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" podStartSLOduration=1.877517726 podStartE2EDuration="2.784357151s" totalImagesPullingTime="906.839425ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:28.782536607 +0000 UTC m=+2057.373822094" watchObservedRunningTime="2026-10-30 22:12:28.784357151 +0000 UTC m=+2057.375642639" Oct 30 22:12:34.658505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:12:34.658475 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:12:49.807232 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:12:49.807190 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:13:19.805875 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:13:19.805835 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:13:24.649029 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.648989 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk"] Oct 30 22:13:24.652904 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.652881 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.707686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.707635 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5d2b13-openshift-default-dockercfg-8ltfq\"" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718251 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p2kn\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-kube-api-access-4p2kn\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718333 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-credential-socket\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718375 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-podinfo\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718527 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-envoy\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718568 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-socket\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718610 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istiod-ca-cert\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718639 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-token\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718695 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-certs\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.719393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.718718 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-data\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.769891 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.769858 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk"] Oct 30 22:13:24.819866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.819835 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-envoy\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.819875 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-socket\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.819903 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istiod-ca-cert\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.819919 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-token\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.819947 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-certs\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.819973 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-data\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.820041 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p2kn\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-kube-api-access-4p2kn\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.820084 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-credential-socket\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.820122 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-podinfo\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820360 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.820332 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-socket\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.820367 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-certs\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820720 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.820696 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-credential-socket\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.820821 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.820706 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-data\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.821027 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.821003 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istiod-ca-cert\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.822286 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.822260 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-envoy\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.822665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.822649 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-podinfo\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.887623 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.887592 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-token\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.887811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.887790 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p2kn\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-kube-api-access-4p2kn\") pod \"e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:24.973700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:24.973610 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:25.411985 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:25.411948 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk"] Oct 30 22:13:25.417907 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:13:25.417877 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd4804f_6ea9_4033_a077_e9e5a246bcd6.slice/crio-1689e6e715420a83b339424654851639c59dff94b574686c17765e10a1f8361d WatchSource:0}: Error finding container 1689e6e715420a83b339424654851639c59dff94b574686c17765e10a1f8361d: Status 404 returned error can't find the container with id 1689e6e715420a83b339424654851639c59dff94b574686c17765e10a1f8361d Oct 30 22:13:25.420558 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:25.420515 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:25.420656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:25.420594 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:25.420656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:25.420630 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:25.973619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:25.973582 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" event={"ID":"1cd4804f-6ea9-4033-a077-e9e5a246bcd6","Type":"ContainerStarted","Data":"a33cbea98b2930236368fa8e77612d10e5d3b15b9aa6275bd9968f8391fc3c7c"} Oct 30 22:13:25.974031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:25.973640 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" event={"ID":"1cd4804f-6ea9-4033-a077-e9e5a246bcd6","Type":"ContainerStarted","Data":"1689e6e715420a83b339424654851639c59dff94b574686c17765e10a1f8361d"} Oct 30 22:13:25.974031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:25.973674 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:26.098896 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:26.098827 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podStartSLOduration=2.098806877 podStartE2EDuration="2.098806877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:26.095677782 +0000 UTC m=+2114.686963270" watchObservedRunningTime="2026-10-30 22:13:26.098806877 +0000 UTC m=+2114.690092367" Oct 30 22:13:26.979963 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:26.979934 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:27.984384 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:27.984353 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:27.985508 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:27.985489 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:13:49.807907 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:13:49.807860 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:13:57.392133 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.392097 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2"] Oct 30 22:13:57.396218 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.396194 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.445200 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445156 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-socket\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.445885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445221 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istiod-ca-cert\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.445885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445262 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-envoy\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.445885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445291 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-podinfo\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.445885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445379 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p24zv\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-kube-api-access-p24zv\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.445885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445450 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-token\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.445885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445503 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-certs\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.445885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445590 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-credential-socket\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.446517 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.445702 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-data\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.462962 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.459491 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-fe745d-openshift-default-dockercfg-rqtq8\"" Oct 30 22:13:57.519078 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.519043 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2"] Oct 30 22:13:57.547722 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.547686 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-certs\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.547921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.547776 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-credential-socket\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.547921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.547864 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-data\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.547921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.547904 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-socket\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.548100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.547933 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istiod-ca-cert\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.548100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.547969 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-envoy\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.548100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.547998 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-podinfo\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.548100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.548083 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p24zv\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-kube-api-access-p24zv\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.548311 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.548134 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-token\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.548311 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.548135 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-certs\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.548488 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.548463 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-socket\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.548563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.548516 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-data\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.549262 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.549232 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istiod-ca-cert\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.551881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.551813 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-envoy\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.553724 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.553697 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-credential-socket\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.557753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.557729 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-podinfo\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.607309 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.607272 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-token\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.607602 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.607567 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p24zv\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-kube-api-access-p24zv\") pod \"e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:57.711091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:57.711005 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:58.140371 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:58.140334 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2"] Oct 30 22:13:58.146403 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:13:58.146373 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f629672_9f9b_41da_b2ff_9bb6fd30d6c3.slice/crio-041cc265a418b03f02cb003161a28c5ba175c565fc87b80051e09c981ed4697b WatchSource:0}: Error finding container 041cc265a418b03f02cb003161a28c5ba175c565fc87b80051e09c981ed4697b: Status 404 returned error can't find the container with id 041cc265a418b03f02cb003161a28c5ba175c565fc87b80051e09c981ed4697b Oct 30 22:13:58.148993 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:58.148956 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:58.149067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:58.149029 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:58.149067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:58.149059 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:13:58.158473 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:58.158444 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" event={"ID":"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3","Type":"ContainerStarted","Data":"041cc265a418b03f02cb003161a28c5ba175c565fc87b80051e09c981ed4697b"} Oct 30 22:13:59.165709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:59.165671 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" event={"ID":"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3","Type":"ContainerStarted","Data":"fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61"} Oct 30 22:13:59.326788 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:59.326734 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" podStartSLOduration=2.326719963 podStartE2EDuration="2.326719963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:59.325066408 +0000 UTC m=+2147.916351894" watchObservedRunningTime="2026-10-30 22:13:59.326719963 +0000 UTC m=+2147.918005449" Oct 30 22:13:59.712062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:59.712025 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:13:59.717769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:13:59.717741 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:14:00.170964 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:00.170930 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:14:00.171990 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:00.171971 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:14:19.805346 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:14:19.805301 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:14:49.804807 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:14:49.804755 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:14:57.776641 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.776606 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8"] Oct 30 22:14:57.780610 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.780589 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.830403 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.830376 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-138566-serving-cert\"" Oct 30 22:14:57.880829 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.880795 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8"] Oct 30 22:14:57.884102 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.884068 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.884243 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.884224 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5aec13-d71b-463d-aecc-1ebdffad74b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.884448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.884395 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626jl\" (UniqueName: \"kubernetes.io/projected/6c5aec13-d71b-463d-aecc-1ebdffad74b2-kube-api-access-626jl\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.884500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.884477 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.985276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.985237 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5aec13-d71b-463d-aecc-1ebdffad74b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.985505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.985349 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626jl\" (UniqueName: \"kubernetes.io/projected/6c5aec13-d71b-463d-aecc-1ebdffad74b2-kube-api-access-626jl\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.985505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.985371 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.985505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.985425 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.985676 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:14:57.985609 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-serving-cert: secret "maas-api-e2e-ipp-b-138566-serving-cert" not found Oct 30 22:14:57.985732 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:14:57.985702 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls podName:6c5aec13-d71b-463d-aecc-1ebdffad74b2 nodeName:}" failed. No retries permitted until 2026-10-30 22:14:58.485677502 +0000 UTC m=+2207.076962985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls") pod "maas-api-e2e-ipp-b-138566-555fc744-qc9n8" (UID: "6c5aec13-d71b-463d-aecc-1ebdffad74b2") : secret "maas-api-e2e-ipp-b-138566-serving-cert" not found Oct 30 22:14:57.986000 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.985979 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5aec13-d71b-463d-aecc-1ebdffad74b2-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:57.988118 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:57.988084 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-metrics-certs\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:58.048696 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:58.048610 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626jl\" (UniqueName: \"kubernetes.io/projected/6c5aec13-d71b-463d-aecc-1ebdffad74b2-kube-api-access-626jl\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:58.493464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:58.493434 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:58.496122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:58.496091 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls\") pod \"maas-api-e2e-ipp-b-138566-555fc744-qc9n8\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:58.694932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:58.694899 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:14:59.075760 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:59.075672 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8"] Oct 30 22:14:59.081468 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:14:59.081438 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5aec13_d71b_463d_aecc_1ebdffad74b2.slice/crio-89d32350250bc05066739beb76ed7fae4b2d8ff8edd65f0bb595ba65720b99b1 WatchSource:0}: Error finding container 89d32350250bc05066739beb76ed7fae4b2d8ff8edd65f0bb595ba65720b99b1: Status 404 returned error can't find the container with id 89d32350250bc05066739beb76ed7fae4b2d8ff8edd65f0bb595ba65720b99b1 Oct 30 22:14:59.511720 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:14:59.511683 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" event={"ID":"6c5aec13-d71b-463d-aecc-1ebdffad74b2","Type":"ContainerStarted","Data":"89d32350250bc05066739beb76ed7fae4b2d8ff8edd65f0bb595ba65720b99b1"} Oct 30 22:15:00.334168 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.334076 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn"] Oct 30 22:15:00.338377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.338346 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:00.370928 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.370890 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7"] Oct 30 22:15:00.374901 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.374881 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:00.395681 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.395655 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\"" Oct 30 22:15:00.417702 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.417453 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3e892-a9e0-4bea-a612-f1ede9d3373e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn\" (UID: \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:00.417702 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.417621 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7\" (UID: \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:00.417702 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.417675 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvqc2\" (UniqueName: \"kubernetes.io/projected/46c3e892-a9e0-4bea-a612-f1ede9d3373e-kube-api-access-tvqc2\") pod \"maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn\" (UID: \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:00.418281 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.417844 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd8f2\" (UniqueName: \"kubernetes.io/projected/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-kube-api-access-dd8f2\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7\" (UID: \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:00.420026 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.420003 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn"] Oct 30 22:15:00.442995 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.442931 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7"] Oct 30 22:15:00.518672 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.518635 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7\" (UID: \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:00.518876 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.518690 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvqc2\" (UniqueName: \"kubernetes.io/projected/46c3e892-a9e0-4bea-a612-f1ede9d3373e-kube-api-access-tvqc2\") pod \"maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn\" (UID: \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:00.518876 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.518778 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd8f2\" (UniqueName: \"kubernetes.io/projected/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-kube-api-access-dd8f2\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7\" (UID: \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:00.519002 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.518887 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3e892-a9e0-4bea-a612-f1ede9d3373e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn\" (UID: \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:00.519602 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.519572 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7\" (UID: \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:00.519729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.519683 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3e892-a9e0-4bea-a612-f1ede9d3373e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn\" (UID: \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:00.521889 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.521866 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" event={"ID":"6c5aec13-d71b-463d-aecc-1ebdffad74b2","Type":"ContainerStarted","Data":"7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4"} Oct 30 22:15:00.521981 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.521918 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:15:00.584898 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.584823 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd8f2\" (UniqueName: \"kubernetes.io/projected/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-kube-api-access-dd8f2\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7\" (UID: \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:00.587997 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.587970 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvqc2\" (UniqueName: \"kubernetes.io/projected/46c3e892-a9e0-4bea-a612-f1ede9d3373e-kube-api-access-tvqc2\") pod \"maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn\" (UID: \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:00.654316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.654282 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:00.688822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.688790 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:00.702036 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:00.701976 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" podStartSLOduration=2.785253569 podStartE2EDuration="3.701956246s" totalImagesPullingTime="916.702677ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:00.69932652 +0000 UTC m=+2209.290612008" watchObservedRunningTime="2026-10-30 22:15:00.701956246 +0000 UTC m=+2209.293241733" Oct 30 22:15:01.035102 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:01.035071 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn"] Oct 30 22:15:01.080264 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:15:01.080187 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c3e892_a9e0_4bea_a612_f1ede9d3373e.slice/crio-258705a2478faf3a0fcd481531da047de87c51c3fa4c8b9d9ea323f88fe0491c WatchSource:0}: Error finding container 258705a2478faf3a0fcd481531da047de87c51c3fa4c8b9d9ea323f88fe0491c: Status 404 returned error can't find the container with id 258705a2478faf3a0fcd481531da047de87c51c3fa4c8b9d9ea323f88fe0491c Oct 30 22:15:01.093597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:01.093571 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7"] Oct 30 22:15:01.099484 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:15:01.099457 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac06a7d6_22f0_4c88_ba1d_4422dfb8aae4.slice/crio-c1db0d1fee260f12ceecfa624c3334c8da14b352dbddaa5e191ad55967939b29 WatchSource:0}: Error finding container c1db0d1fee260f12ceecfa624c3334c8da14b352dbddaa5e191ad55967939b29: Status 404 returned error can't find the container with id c1db0d1fee260f12ceecfa624c3334c8da14b352dbddaa5e191ad55967939b29 Oct 30 22:15:01.528214 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:01.528170 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" event={"ID":"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4","Type":"ContainerStarted","Data":"c1db0d1fee260f12ceecfa624c3334c8da14b352dbddaa5e191ad55967939b29"} Oct 30 22:15:01.529673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:01.529640 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" event={"ID":"46c3e892-a9e0-4bea-a612-f1ede9d3373e","Type":"ContainerStarted","Data":"258705a2478faf3a0fcd481531da047de87c51c3fa4c8b9d9ea323f88fe0491c"} Oct 30 22:15:05.559744 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:05.559706 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" event={"ID":"46c3e892-a9e0-4bea-a612-f1ede9d3373e","Type":"ContainerStarted","Data":"a938411e8021a036b08bd4e1ecc2999f4e6aba32773cb1278e35512e327c1e75"} Oct 30 22:15:05.561443 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:05.561389 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" event={"ID":"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4","Type":"ContainerStarted","Data":"e8276a0bd9177399b8e70d2bf50a68c64ce70991fe92424ccec3ce3719dfb41f"} Oct 30 22:15:05.672636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:05.672522 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" podStartSLOduration=1.670152242 podStartE2EDuration="5.672503606s" totalImagesPullingTime="4.002351364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.670649677 +0000 UTC m=+2214.261935165" watchObservedRunningTime="2026-10-30 22:15:05.672503606 +0000 UTC m=+2214.263789094" Oct 30 22:15:05.782769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:05.782715 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" podStartSLOduration=1.795252979 podStartE2EDuration="5.782702276s" totalImagesPullingTime="3.987449297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.779921727 +0000 UTC m=+2214.371207216" watchObservedRunningTime="2026-10-30 22:15:05.782702276 +0000 UTC m=+2214.373987781" Oct 30 22:15:06.543440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:06.542409 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:15:06.575934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:06.575897 5139 generic.go:356] "Generic (PLEG): container finished" podID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerID="a938411e8021a036b08bd4e1ecc2999f4e6aba32773cb1278e35512e327c1e75" exitCode=60 Oct 30 22:15:06.576619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:06.576591 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" event={"ID":"46c3e892-a9e0-4bea-a612-f1ede9d3373e","Type":"ContainerDied","Data":"a938411e8021a036b08bd4e1ecc2999f4e6aba32773cb1278e35512e327c1e75"} Oct 30 22:15:06.577251 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:06.577224 5139 scope.go:122] "RemoveContainer" containerID="a938411e8021a036b08bd4e1ecc2999f4e6aba32773cb1278e35512e327c1e75" Oct 30 22:15:06.586446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:06.583349 5139 generic.go:356] "Generic (PLEG): container finished" podID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerID="e8276a0bd9177399b8e70d2bf50a68c64ce70991fe92424ccec3ce3719dfb41f" exitCode=60 Oct 30 22:15:06.586446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:06.583404 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" event={"ID":"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4","Type":"ContainerDied","Data":"e8276a0bd9177399b8e70d2bf50a68c64ce70991fe92424ccec3ce3719dfb41f"} Oct 30 22:15:06.586446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:06.583828 5139 scope.go:122] "RemoveContainer" containerID="e8276a0bd9177399b8e70d2bf50a68c64ce70991fe92424ccec3ce3719dfb41f" Oct 30 22:15:07.616048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:07.615997 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" event={"ID":"46c3e892-a9e0-4bea-a612-f1ede9d3373e","Type":"ContainerStarted","Data":"5327c176e6b809cd1f04ef579d812212f37728962939d608ffb7d046e63d6bde"} Oct 30 22:15:07.616674 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:07.616644 5139 scope.go:122] "RemoveContainer" containerID="5327c176e6b809cd1f04ef579d812212f37728962939d608ffb7d046e63d6bde" Oct 30 22:15:07.617271 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:07.617237 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn_odh-ai-gateway-infra(46c3e892-a9e0-4bea-a612-f1ede9d3373e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" Oct 30 22:15:07.620489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:07.620465 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" event={"ID":"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4","Type":"ContainerStarted","Data":"224cbf6aed8295e464a87ee99a8f56c85e3684d46e4c9ed83a7cfbbb62aa77cf"} Oct 30 22:15:08.627564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:08.627528 5139 generic.go:356] "Generic (PLEG): container finished" podID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerID="224cbf6aed8295e464a87ee99a8f56c85e3684d46e4c9ed83a7cfbbb62aa77cf" exitCode=60 Oct 30 22:15:08.628046 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:08.627569 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" event={"ID":"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4","Type":"ContainerDied","Data":"224cbf6aed8295e464a87ee99a8f56c85e3684d46e4c9ed83a7cfbbb62aa77cf"} Oct 30 22:15:08.628046 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:08.627630 5139 scope.go:122] "RemoveContainer" containerID="e8276a0bd9177399b8e70d2bf50a68c64ce70991fe92424ccec3ce3719dfb41f" Oct 30 22:15:08.628046 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:08.627884 5139 scope.go:122] "RemoveContainer" containerID="224cbf6aed8295e464a87ee99a8f56c85e3684d46e4c9ed83a7cfbbb62aa77cf" Oct 30 22:15:08.628205 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:08.628179 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7_odh-ai-gateway-infra(ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" Oct 30 22:15:08.629873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:08.629851 5139 generic.go:356] "Generic (PLEG): container finished" podID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerID="5327c176e6b809cd1f04ef579d812212f37728962939d608ffb7d046e63d6bde" exitCode=60 Oct 30 22:15:08.629955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:08.629882 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" event={"ID":"46c3e892-a9e0-4bea-a612-f1ede9d3373e","Type":"ContainerDied","Data":"5327c176e6b809cd1f04ef579d812212f37728962939d608ffb7d046e63d6bde"} Oct 30 22:15:08.630176 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:08.630161 5139 scope.go:122] "RemoveContainer" containerID="5327c176e6b809cd1f04ef579d812212f37728962939d608ffb7d046e63d6bde" Oct 30 22:15:08.630441 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:08.630399 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn_odh-ai-gateway-infra(46c3e892-a9e0-4bea-a612-f1ede9d3373e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" Oct 30 22:15:08.639209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:08.639192 5139 scope.go:122] "RemoveContainer" containerID="a938411e8021a036b08bd4e1ecc2999f4e6aba32773cb1278e35512e327c1e75" Oct 30 22:15:09.636583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:09.636553 5139 scope.go:122] "RemoveContainer" containerID="224cbf6aed8295e464a87ee99a8f56c85e3684d46e4c9ed83a7cfbbb62aa77cf" Oct 30 22:15:09.637043 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:09.636884 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7_odh-ai-gateway-infra(ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" Oct 30 22:15:09.638179 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:09.638157 5139 scope.go:122] "RemoveContainer" containerID="5327c176e6b809cd1f04ef579d812212f37728962939d608ffb7d046e63d6bde" Oct 30 22:15:09.638461 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:09.638439 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn_odh-ai-gateway-infra(46c3e892-a9e0-4bea-a612-f1ede9d3373e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" Oct 30 22:15:14.025579 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:14.025533 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:15:14.025998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:14.025635 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:15:16.688654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:16.688618 5139 generic.go:356] "Generic (PLEG): container finished" podID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerID="a33cbea98b2930236368fa8e77612d10e5d3b15b9aa6275bd9968f8391fc3c7c" exitCode=0 Oct 30 22:15:16.689132 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:16.688683 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" event={"ID":"1cd4804f-6ea9-4033-a077-e9e5a246bcd6","Type":"ContainerDied","Data":"a33cbea98b2930236368fa8e77612d10e5d3b15b9aa6275bd9968f8391fc3c7c"} Oct 30 22:15:16.689132 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:16.689103 5139 scope.go:122] "RemoveContainer" containerID="a33cbea98b2930236368fa8e77612d10e5d3b15b9aa6275bd9968f8391fc3c7c" Oct 30 22:15:16.690682 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:16.690654 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:16.690743 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:16.690726 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:16.690784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:16.690759 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:17.696572 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:17.696533 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" event={"ID":"1cd4804f-6ea9-4033-a077-e9e5a246bcd6","Type":"ContainerStarted","Data":"016abd1ed7520f6bd4932753902f3dfcc61a87b6b14473d84db32e7c3b4a1960"} Oct 30 22:15:17.987502 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:17.983780 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:15:17.987502 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:17.983925 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:15:17.987502 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:17.983969 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 22:15:18.984847 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:18.984809 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:15:18.985479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:18.985445 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 22:15:19.815951 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:19.815912 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:15:19.986440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:19.983260 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:15:19.986440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:19.983337 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 22:15:20.987960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:20.987912 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:15:20.988537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:20.988006 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 22:15:21.539451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:21.539406 5139 scope.go:122] "RemoveContainer" containerID="5327c176e6b809cd1f04ef579d812212f37728962939d608ffb7d046e63d6bde" Oct 30 22:15:21.539660 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:21.539582 5139 scope.go:122] "RemoveContainer" containerID="224cbf6aed8295e464a87ee99a8f56c85e3684d46e4c9ed83a7cfbbb62aa77cf" Oct 30 22:15:21.975859 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:21.975828 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 22:15:21.976085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:21.976058 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 22:15:22.726771 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.726738 5139 generic.go:356] "Generic (PLEG): container finished" podID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerID="6b35a3b922e8c0e463727cced150375539c887cb074988dd480d9d6334d90e4a" exitCode=60 Oct 30 22:15:22.727246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.726807 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" event={"ID":"46c3e892-a9e0-4bea-a612-f1ede9d3373e","Type":"ContainerDied","Data":"6b35a3b922e8c0e463727cced150375539c887cb074988dd480d9d6334d90e4a"} Oct 30 22:15:22.727246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.726866 5139 scope.go:122] "RemoveContainer" containerID="5327c176e6b809cd1f04ef579d812212f37728962939d608ffb7d046e63d6bde" Oct 30 22:15:22.727400 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.727345 5139 scope.go:122] "RemoveContainer" containerID="6b35a3b922e8c0e463727cced150375539c887cb074988dd480d9d6334d90e4a" Oct 30 22:15:22.727766 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:22.727738 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn_odh-ai-gateway-infra(46c3e892-a9e0-4bea-a612-f1ede9d3373e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" Oct 30 22:15:22.729389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.729366 5139 generic.go:356] "Generic (PLEG): container finished" podID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerID="0d33d51471291d2db1cad983bec0688a5d03bea62a15e74209c5469bbb211f17" exitCode=60 Oct 30 22:15:22.729496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.729454 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" event={"ID":"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4","Type":"ContainerDied","Data":"0d33d51471291d2db1cad983bec0688a5d03bea62a15e74209c5469bbb211f17"} Oct 30 22:15:22.729813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.729799 5139 scope.go:122] "RemoveContainer" containerID="0d33d51471291d2db1cad983bec0688a5d03bea62a15e74209c5469bbb211f17" Oct 30 22:15:22.730116 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:22.730095 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7_odh-ai-gateway-infra(ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" Oct 30 22:15:22.802315 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.802290 5139 scope.go:122] "RemoveContainer" containerID="224cbf6aed8295e464a87ee99a8f56c85e3684d46e4c9ed83a7cfbbb62aa77cf" Oct 30 22:15:22.986189 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.986101 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:15:22.986826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.986797 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:15:22.988841 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:22.988822 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:15:23.860906 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:23.860867 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn"] Oct 30 22:15:23.962667 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:23.962631 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7"] Oct 30 22:15:24.024344 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.023863 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:24.108527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.108503 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:24.166290 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.166197 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c3e892-a9e0-4bea-a612-f1ede9d3373e-kube-api-access-tvqc2\" (UniqueName: \"kubernetes.io/projected/46c3e892-a9e0-4bea-a612-f1ede9d3373e-kube-api-access-tvqc2\") pod \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\" (UID: \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\") " Oct 30 22:15:24.166515 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.166292 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-kube-api-access-dd8f2\" (UniqueName: \"kubernetes.io/projected/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-kube-api-access-dd8f2\") pod \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\" (UID: \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\") " Oct 30 22:15:24.166515 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.166327 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-maas-api-service-ca\") pod \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\" (UID: \"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4\") " Oct 30 22:15:24.166648 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.166600 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c3e892-a9e0-4bea-a612-f1ede9d3373e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3e892-a9e0-4bea-a612-f1ede9d3373e-maas-api-service-ca\") pod \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\" (UID: \"46c3e892-a9e0-4bea-a612-f1ede9d3373e\") " Oct 30 22:15:24.166805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.166777 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-maas-api-service-ca" pod "ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" (UID: "ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:24.167017 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.166960 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c3e892-a9e0-4bea-a612-f1ede9d3373e-maas-api-service-ca" pod "46c3e892-a9e0-4bea-a612-f1ede9d3373e" (UID: "46c3e892-a9e0-4bea-a612-f1ede9d3373e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:15:24.167101 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.167091 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c3e892-a9e0-4bea-a612-f1ede9d3373e-maas-api-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.167166 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.167105 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-maas-api-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.168509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.168485 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c3e892-a9e0-4bea-a612-f1ede9d3373e-kube-api-access-tvqc2" pod "46c3e892-a9e0-4bea-a612-f1ede9d3373e" (UID: "46c3e892-a9e0-4bea-a612-f1ede9d3373e"). InnerVolumeSpecName "kube-api-access-tvqc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:24.168619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.168605 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-kube-api-access-dd8f2" pod "ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" (UID: "ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4"). InnerVolumeSpecName "kube-api-access-dd8f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:15:24.268211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.268176 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd8f2\" (UniqueName: \"kubernetes.io/projected/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4-kube-api-access-dd8f2\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.268211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.268210 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvqc2\" (UniqueName: \"kubernetes.io/projected/46c3e892-a9e0-4bea-a612-f1ede9d3373e-kube-api-access-tvqc2\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:15:24.745022 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.744988 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" Oct 30 22:15:24.745232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.745016 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn" event={"ID":"46c3e892-a9e0-4bea-a612-f1ede9d3373e","Type":"ContainerDied","Data":"258705a2478faf3a0fcd481531da047de87c51c3fa4c8b9d9ea323f88fe0491c"} Oct 30 22:15:24.745232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.745090 5139 scope.go:122] "RemoveContainer" containerID="6b35a3b922e8c0e463727cced150375539c887cb074988dd480d9d6334d90e4a" Oct 30 22:15:24.746470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.746448 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" Oct 30 22:15:24.746578 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.746467 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7" event={"ID":"ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4","Type":"ContainerDied","Data":"c1db0d1fee260f12ceecfa624c3334c8da14b352dbddaa5e191ad55967939b29"} Oct 30 22:15:24.756523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.756408 5139 scope.go:122] "RemoveContainer" containerID="0d33d51471291d2db1cad983bec0688a5d03bea62a15e74209c5469bbb211f17" Oct 30 22:15:24.900869 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.900829 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7"] Oct 30 22:15:24.905753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:24.905727 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889975-6mnh7"] Oct 30 22:15:25.008174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:25.008103 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn"] Oct 30 22:15:25.012577 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:25.012549 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-95b84c-29889975-hbnsn"] Oct 30 22:15:25.542778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:25.542741 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" path="/var/lib/kubelet/pods/46c3e892-a9e0-4bea-a612-f1ede9d3373e/volumes" Oct 30 22:15:25.543223 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:25.543208 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" path="/var/lib/kubelet/pods/ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4/volumes" Oct 30 22:15:49.824132 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:15:49.824091 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:15:57.944863 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:57.944758 5139 generic.go:356] "Generic (PLEG): container finished" podID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerID="fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61" exitCode=0 Oct 30 22:15:57.944863 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:57.944838 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" event={"ID":"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3","Type":"ContainerDied","Data":"fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61"} Oct 30 22:15:57.945336 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:57.945231 5139 scope.go:122] "RemoveContainer" containerID="fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61" Oct 30 22:15:57.946877 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:57.946847 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:57.946955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:57.946917 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:57.946991 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:57.946953 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:15:58.953709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:58.953671 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" event={"ID":"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3","Type":"ContainerStarted","Data":"c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8"} Oct 30 22:15:59.711625 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:59.711585 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:15:59.712732 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:59.712707 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:15:59.712885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:15:59.712752 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:16:00.187444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:00.183524 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:16:00.712324 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:00.712287 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:16:00.712535 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:00.712339 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:16:01.711840 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:01.711803 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:16:01.712261 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:01.711866 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:16:02.711630 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:02.711595 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:16:02.711816 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:02.711660 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:16:03.711510 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:03.711476 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 22:16:03.711936 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:03.711530 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 22:16:04.716595 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:04.716561 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:16:04.717673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:04.717653 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:16:19.805599 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:16:19.805562 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:16:49.805808 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:16:49.805765 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:16:59.845465 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:59.845429 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt"] Oct 30 22:16:59.845909 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:16:59.845643 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" podUID="a4082918-3ac0-4ca5-aa66-2f3dee8969ab" containerName="maas-api" containerID="cri-o://a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9" gracePeriod=30 Oct 30 22:17:00.118360 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.118335 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:17:00.189982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.189949 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-kube-api-access-l7w8h\" (UniqueName: \"kubernetes.io/projected/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-kube-api-access-l7w8h\") pod \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " Oct 30 22:17:00.190159 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.190093 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-openshift-service-ca\") pod \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " Oct 30 22:17:00.190159 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.190118 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-metrics-certs\") pod \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " Oct 30 22:17:00.190293 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.190159 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls\") pod \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\" (UID: \"a4082918-3ac0-4ca5-aa66-2f3dee8969ab\") " Oct 30 22:17:00.190558 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.190521 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-openshift-service-ca" pod "a4082918-3ac0-4ca5-aa66-2f3dee8969ab" (UID: "a4082918-3ac0-4ca5-aa66-2f3dee8969ab"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:17:00.190715 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.190681 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:00.192516 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.192490 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-kube-api-access-l7w8h" pod "a4082918-3ac0-4ca5-aa66-2f3dee8969ab" (UID: "a4082918-3ac0-4ca5-aa66-2f3dee8969ab"). InnerVolumeSpecName "kube-api-access-l7w8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:17:00.192625 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.192575 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls" pod "a4082918-3ac0-4ca5-aa66-2f3dee8969ab" (UID: "a4082918-3ac0-4ca5-aa66-2f3dee8969ab"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:00.192684 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.192655 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-metrics-certs" pod "a4082918-3ac0-4ca5-aa66-2f3dee8969ab" (UID: "a4082918-3ac0-4ca5-aa66-2f3dee8969ab"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:17:00.291920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.291881 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:00.291920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.291915 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:00.291920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.291926 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7w8h\" (UniqueName: \"kubernetes.io/projected/a4082918-3ac0-4ca5-aa66-2f3dee8969ab-kube-api-access-l7w8h\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:17:00.313628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.313588 5139 generic.go:356] "Generic (PLEG): container finished" podID="a4082918-3ac0-4ca5-aa66-2f3dee8969ab" containerID="a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9" exitCode=0 Oct 30 22:17:00.313769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.313646 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" Oct 30 22:17:00.313769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.313676 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" event={"ID":"a4082918-3ac0-4ca5-aa66-2f3dee8969ab","Type":"ContainerDied","Data":"a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9"} Oct 30 22:17:00.313769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.313731 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt" event={"ID":"a4082918-3ac0-4ca5-aa66-2f3dee8969ab","Type":"ContainerDied","Data":"2094cb6cb164fce4f9cd74b4bd0d96b9cf37ebe6b1256c4f5a381cfe71f3a7f8"} Oct 30 22:17:00.313769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.313760 5139 scope.go:122] "RemoveContainer" containerID="a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9" Oct 30 22:17:00.323726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.323709 5139 scope.go:122] "RemoveContainer" containerID="a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9" Oct 30 22:17:00.323994 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:17:00.323970 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9\": container with ID starting with a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9 not found: ID does not exist" containerID="a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9" Oct 30 22:17:00.324059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.324002 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9"} err="failed to get container status \"a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9\": rpc error: code = NotFound desc = could not find container \"a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9\": container with ID starting with a4109cb6982cf840235775f4c85245719e073dd516daca6f4f02ac654eb2b7b9 not found: ID does not exist" Oct 30 22:17:00.467888 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.467800 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt"] Oct 30 22:17:00.475663 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:00.474370 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-a3aa65-67849d9545-5njxt"] Oct 30 22:17:01.543604 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:01.543568 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4082918-3ac0-4ca5-aa66-2f3dee8969ab" path="/var/lib/kubelet/pods/a4082918-3ac0-4ca5-aa66-2f3dee8969ab/volumes" Oct 30 22:17:19.806578 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:17:19.806530 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:17:44.382460 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:44.382403 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ccz7d_9bbdcb9f-b9c8-46b3-8925-7c246f2affe3/payload-processing/0.log" Oct 30 22:17:44.656525 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:17:44.656439 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ccz7d_9bbdcb9f-b9c8-46b3-8925-7c246f2affe3/payload-processing/0.log" Oct 30 22:17:49.805034 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:17:49.804990 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:18:17.742908 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:17.742871 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb"] Oct 30 22:18:17.743500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:17.743094 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" podUID="ac3b2a5a-8e7d-4131-9354-4aac1f436c08" containerName="maas-api" containerID="cri-o://476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9" gracePeriod=30 Oct 30 22:18:18.054085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.054056 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:18:18.124697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.124659 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-openshift-service-ca\") pod \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " Oct 30 22:18:18.124884 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.124764 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-kube-api-access-72p2r\" (UniqueName: \"kubernetes.io/projected/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-kube-api-access-72p2r\") pod \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " Oct 30 22:18:18.124884 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.124824 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-metrics-certs\") pod \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " Oct 30 22:18:18.124987 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.124907 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls\") pod \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\" (UID: \"ac3b2a5a-8e7d-4131-9354-4aac1f436c08\") " Oct 30 22:18:18.125066 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.125032 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-openshift-service-ca" pod "ac3b2a5a-8e7d-4131-9354-4aac1f436c08" (UID: "ac3b2a5a-8e7d-4131-9354-4aac1f436c08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:18.125394 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.125353 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:18.127533 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.127502 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls" pod "ac3b2a5a-8e7d-4131-9354-4aac1f436c08" (UID: "ac3b2a5a-8e7d-4131-9354-4aac1f436c08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:18.127645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.127608 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-metrics-certs" pod "ac3b2a5a-8e7d-4131-9354-4aac1f436c08" (UID: "ac3b2a5a-8e7d-4131-9354-4aac1f436c08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:18.127742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.127726 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-kube-api-access-72p2r" pod "ac3b2a5a-8e7d-4131-9354-4aac1f436c08" (UID: "ac3b2a5a-8e7d-4131-9354-4aac1f436c08"). InnerVolumeSpecName "kube-api-access-72p2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:18.226042 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.226001 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72p2r\" (UniqueName: \"kubernetes.io/projected/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-kube-api-access-72p2r\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:18.226042 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.226039 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:18.226042 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.226048 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ac3b2a5a-8e7d-4131-9354-4aac1f436c08-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:18.768629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.768594 5139 generic.go:356] "Generic (PLEG): container finished" podID="ac3b2a5a-8e7d-4131-9354-4aac1f436c08" containerID="476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9" exitCode=0 Oct 30 22:18:18.769067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.768637 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" Oct 30 22:18:18.769067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.768683 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" event={"ID":"ac3b2a5a-8e7d-4131-9354-4aac1f436c08","Type":"ContainerDied","Data":"476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9"} Oct 30 22:18:18.769067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.768735 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb" event={"ID":"ac3b2a5a-8e7d-4131-9354-4aac1f436c08","Type":"ContainerDied","Data":"c9f1fdf73916f3a78ac4712aac43b7bc41bc0c9b13fc45cff877af29bcc85c16"} Oct 30 22:18:18.769067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.768765 5139 scope.go:122] "RemoveContainer" containerID="476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9" Oct 30 22:18:18.780702 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.780673 5139 scope.go:122] "RemoveContainer" containerID="476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9" Oct 30 22:18:18.781263 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:18:18.781233 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9\": container with ID starting with 476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9 not found: ID does not exist" containerID="476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9" Oct 30 22:18:18.781442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.781273 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9"} err="failed to get container status \"476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9\": rpc error: code = NotFound desc = could not find container \"476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9\": container with ID starting with 476b126883b3dcb13c71f52df4ba1102a02464f9fd8c7be90a61468b96825cc9 not found: ID does not exist" Oct 30 22:18:18.939594 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.935250 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb"] Oct 30 22:18:18.939975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:18.939941 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-95b84c-5c7f5ffc49-vctfb"] Oct 30 22:18:19.544091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:19.544046 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3b2a5a-8e7d-4131-9354-4aac1f436c08" path="/var/lib/kubelet/pods/ac3b2a5a-8e7d-4131-9354-4aac1f436c08/volumes" Oct 30 22:18:19.836477 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:18:19.834821 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:18:41.407721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.407682 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8"] Oct 30 22:18:41.408194 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.407934 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" podUID="6c5aec13-d71b-463d-aecc-1ebdffad74b2" containerName="maas-api" containerID="cri-o://7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4" gracePeriod=30 Oct 30 22:18:41.530569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.530531 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" podUID="6c5aec13-d71b-463d-aecc-1ebdffad74b2" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.90:8443/health\": dial tcp 10.134.0.90:8443: connect: connection refused" Oct 30 22:18:41.703910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.703883 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:18:41.741323 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.741292 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls\") pod \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " Oct 30 22:18:41.741493 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.741449 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-metrics-certs\") pod \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " Oct 30 22:18:41.741581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.741541 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c5aec13-d71b-463d-aecc-1ebdffad74b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5aec13-d71b-463d-aecc-1ebdffad74b2-openshift-service-ca\") pod \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " Oct 30 22:18:41.741581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.741570 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c5aec13-d71b-463d-aecc-1ebdffad74b2-kube-api-access-626jl\" (UniqueName: \"kubernetes.io/projected/6c5aec13-d71b-463d-aecc-1ebdffad74b2-kube-api-access-626jl\") pod \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\" (UID: \"6c5aec13-d71b-463d-aecc-1ebdffad74b2\") " Oct 30 22:18:41.741913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.741886 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c5aec13-d71b-463d-aecc-1ebdffad74b2-openshift-service-ca" pod "6c5aec13-d71b-463d-aecc-1ebdffad74b2" (UID: "6c5aec13-d71b-463d-aecc-1ebdffad74b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:18:41.743448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.743383 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls" pod "6c5aec13-d71b-463d-aecc-1ebdffad74b2" (UID: "6c5aec13-d71b-463d-aecc-1ebdffad74b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:41.743583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.743561 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c5aec13-d71b-463d-aecc-1ebdffad74b2-kube-api-access-626jl" pod "6c5aec13-d71b-463d-aecc-1ebdffad74b2" (UID: "6c5aec13-d71b-463d-aecc-1ebdffad74b2"). InnerVolumeSpecName "kube-api-access-626jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:18:41.743648 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.743608 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-metrics-certs" pod "6c5aec13-d71b-463d-aecc-1ebdffad74b2" (UID: "6c5aec13-d71b-463d-aecc-1ebdffad74b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:18:41.842672 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.842639 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:41.842672 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.842666 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c5aec13-d71b-463d-aecc-1ebdffad74b2-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:41.842672 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.842678 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c5aec13-d71b-463d-aecc-1ebdffad74b2-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:41.842895 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.842687 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626jl\" (UniqueName: \"kubernetes.io/projected/6c5aec13-d71b-463d-aecc-1ebdffad74b2-kube-api-access-626jl\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:18:41.899690 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.899650 5139 generic.go:356] "Generic (PLEG): container finished" podID="6c5aec13-d71b-463d-aecc-1ebdffad74b2" containerID="7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4" exitCode=0 Oct 30 22:18:41.899864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.899696 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" Oct 30 22:18:41.899864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.899739 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" event={"ID":"6c5aec13-d71b-463d-aecc-1ebdffad74b2","Type":"ContainerDied","Data":"7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4"} Oct 30 22:18:41.899864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.899795 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8" event={"ID":"6c5aec13-d71b-463d-aecc-1ebdffad74b2","Type":"ContainerDied","Data":"89d32350250bc05066739beb76ed7fae4b2d8ff8edd65f0bb595ba65720b99b1"} Oct 30 22:18:41.899864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.899823 5139 scope.go:122] "RemoveContainer" containerID="7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4" Oct 30 22:18:41.910331 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.910311 5139 scope.go:122] "RemoveContainer" containerID="7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4" Oct 30 22:18:41.910624 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:18:41.910605 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4\": container with ID starting with 7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4 not found: ID does not exist" containerID="7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4" Oct 30 22:18:41.910681 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:41.910635 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4"} err="failed to get container status \"7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4\": rpc error: code = NotFound desc = could not find container \"7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4\": container with ID starting with 7dd3eded7939b248e316ddb1f2822c66c41aa1d668950c088501869ffc88a2c4 not found: ID does not exist" Oct 30 22:18:42.060018 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:42.059970 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8"] Oct 30 22:18:42.063873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:42.063848 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-138566-555fc744-qc9n8"] Oct 30 22:18:43.546607 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:18:43.546574 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c5aec13-d71b-463d-aecc-1ebdffad74b2" path="/var/lib/kubelet/pods/6c5aec13-d71b-463d-aecc-1ebdffad74b2/volumes" Oct 30 22:18:49.805621 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:18:49.805578 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:19:05.610683 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.610646 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq"] Oct 30 22:19:05.611094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.610859 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" podUID="92b72f34-d257-4fe1-9c9b-6ecab47534bc" containerName="discovery" containerID="cri-o://8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3" gracePeriod=30 Oct 30 22:19:05.880975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.880907 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:19:05.974773 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.974737 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-cacerts\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-cacerts\") pod \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " Oct 30 22:19:05.974773 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.974775 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-kubeconfig\") pod \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " Oct 30 22:19:05.975022 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.974813 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b72f34-d257-4fe1-9c9b-6ecab47534bc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b72f34-d257-4fe1-9c9b-6ecab47534bc-local-certs\") pod \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " Oct 30 22:19:05.975022 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.974851 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-token\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-token\") pod \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " Oct 30 22:19:05.975022 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.974904 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-kube-api-access-gfjxz\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-kube-api-access-gfjxz\") pod \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " Oct 30 22:19:05.975188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.975047 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-ca-configmap\") pod \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " Oct 30 22:19:05.975188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.975075 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-dns-cert\") pod \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\" (UID: \"92b72f34-d257-4fe1-9c9b-6ecab47534bc\") " Oct 30 22:19:05.975511 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.975481 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-ca-configmap" pod "92b72f34-d257-4fe1-9c9b-6ecab47534bc" (UID: "92b72f34-d257-4fe1-9c9b-6ecab47534bc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:05.977255 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.977194 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-cacerts" pod "92b72f34-d257-4fe1-9c9b-6ecab47534bc" (UID: "92b72f34-d257-4fe1-9c9b-6ecab47534bc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:05.977552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.977514 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-kubeconfig" pod "92b72f34-d257-4fe1-9c9b-6ecab47534bc" (UID: "92b72f34-d257-4fe1-9c9b-6ecab47534bc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:05.977701 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.977676 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-token" pod "92b72f34-d257-4fe1-9c9b-6ecab47534bc" (UID: "92b72f34-d257-4fe1-9c9b-6ecab47534bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:05.977748 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.977684 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-kube-api-access-gfjxz" pod "92b72f34-d257-4fe1-9c9b-6ecab47534bc" (UID: "92b72f34-d257-4fe1-9c9b-6ecab47534bc"). InnerVolumeSpecName "kube-api-access-gfjxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:05.977748 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.977721 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b72f34-d257-4fe1-9c9b-6ecab47534bc-local-certs" pod "92b72f34-d257-4fe1-9c9b-6ecab47534bc" (UID: "92b72f34-d257-4fe1-9c9b-6ecab47534bc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:05.977884 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:05.977861 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-dns-cert" pod "92b72f34-d257-4fe1-9c9b-6ecab47534bc" (UID: "92b72f34-d257-4fe1-9c9b-6ecab47534bc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:19:06.076588 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.076558 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:06.076588 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.076587 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfjxz\" (UniqueName: \"kubernetes.io/projected/92b72f34-d257-4fe1-9c9b-6ecab47534bc-kube-api-access-gfjxz\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:06.076789 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.076597 5139 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-ca-configmap\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:06.076789 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.076607 5139 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-csr-dns-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:06.076789 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.076616 5139 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-cacerts\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:06.076789 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.076625 5139 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/92b72f34-d257-4fe1-9c9b-6ecab47534bc-istio-kubeconfig\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:06.076789 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.076633 5139 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b72f34-d257-4fe1-9c9b-6ecab47534bc-local-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:06.079587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.079557 5139 generic.go:356] "Generic (PLEG): container finished" podID="92b72f34-d257-4fe1-9c9b-6ecab47534bc" containerID="8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3" exitCode=0 Oct 30 22:19:06.079711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.079613 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" Oct 30 22:19:06.079711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.079651 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" event={"ID":"92b72f34-d257-4fe1-9c9b-6ecab47534bc","Type":"ContainerDied","Data":"8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3"} Oct 30 22:19:06.079784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.079745 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq" event={"ID":"92b72f34-d257-4fe1-9c9b-6ecab47534bc","Type":"ContainerDied","Data":"61b256ca263fbafdbb26bca13d2e79e5433e835283e691305d9bde8ce3af014e"} Oct 30 22:19:06.079784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.079773 5139 scope.go:122] "RemoveContainer" containerID="8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3" Oct 30 22:19:06.090610 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.090579 5139 scope.go:122] "RemoveContainer" containerID="8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3" Oct 30 22:19:06.090966 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:19:06.090946 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3\": container with ID starting with 8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3 not found: ID does not exist" containerID="8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3" Oct 30 22:19:06.091037 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.090974 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3"} err="failed to get container status \"8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3\": rpc error: code = NotFound desc = could not find container \"8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3\": container with ID starting with 8fe961e71375b0e38d7c46a505536d07e70dbd0a987a3197c8b53fbd6f85dac3 not found: ID does not exist" Oct 30 22:19:06.235125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.235040 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq"] Oct 30 22:19:06.243309 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.243281 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bddq"] Oct 30 22:19:06.995601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.995568 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns"] Oct 30 22:19:06.996056 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:06.995797 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" containerID="cri-o://be7974ad0cfe40ea7fac4d396990d64f2b5a1ed5cec82954502dea621a327059" gracePeriod=30 Oct 30 22:19:07.544397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:07.544361 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b72f34-d257-4fe1-9c9b-6ecab47534bc" path="/var/lib/kubelet/pods/92b72f34-d257-4fe1-9c9b-6ecab47534bc/volumes" Oct 30 22:19:12.119349 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.119327 5139 generic.go:356] "Generic (PLEG): container finished" podID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerID="be7974ad0cfe40ea7fac4d396990d64f2b5a1ed5cec82954502dea621a327059" exitCode=0 Oct 30 22:19:12.119716 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.119409 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" event={"ID":"dd723b38-6cea-4308-bcf3-26369bbfeecc","Type":"ContainerDied","Data":"be7974ad0cfe40ea7fac4d396990d64f2b5a1ed5cec82954502dea621a327059"} Oct 30 22:19:12.119716 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.119532 5139 scope.go:122] "RemoveContainer" containerID="d0091646eb464ed633ffd8710361fc7d45eced785f55eda5876bd9b4c3e896b8" Oct 30 22:19:12.288095 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.288070 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 22:19:12.350721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.350694 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd723b38-6cea-4308-bcf3-26369bbfeecc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd723b38-6cea-4308-bcf3-26369bbfeecc-istiod-ca-cert\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.350886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.350796 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-token\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-token\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.350886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.350830 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-kube-api-access-5ppx9\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-kube-api-access-5ppx9\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.350991 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.350917 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-certs\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.350991 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.350957 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-credential-socket\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.351113 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351091 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-socket\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.351175 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351137 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-podinfo\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.351175 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351164 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-data\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.351283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351203 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-envoy\") pod \"dd723b38-6cea-4308-bcf3-26369bbfeecc\" (UID: \"dd723b38-6cea-4308-bcf3-26369bbfeecc\") " Oct 30 22:19:12.351283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351108 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd723b38-6cea-4308-bcf3-26369bbfeecc-istiod-ca-cert" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:12.351283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351200 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-certs" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.351488 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351302 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-credential-socket" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.351488 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351343 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-socket" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.351488 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351432 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-data" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.351718 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351697 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.351792 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351723 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.351792 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351739 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd723b38-6cea-4308-bcf3-26369bbfeecc-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.351792 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351752 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.351792 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.351766 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.353155 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.353128 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-token" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:12.353298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.353272 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-podinfo" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:12.353399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.353311 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-kube-api-access-5ppx9" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "kube-api-access-5ppx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:12.353399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.353326 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-envoy" pod "dd723b38-6cea-4308-bcf3-26369bbfeecc" (UID: "dd723b38-6cea-4308-bcf3-26369bbfeecc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:12.452762 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.452722 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.452762 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.452759 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.453049 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.452772 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:12.453049 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:12.452785 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ppx9\" (UniqueName: \"kubernetes.io/projected/dd723b38-6cea-4308-bcf3-26369bbfeecc-kube-api-access-5ppx9\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:13.130295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:13.130241 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" event={"ID":"dd723b38-6cea-4308-bcf3-26369bbfeecc","Type":"ContainerDied","Data":"d0531eaf33bbbbb34a562411abf0f9cb059471459219bc5417db8d065e3b9da1"} Oct 30 22:19:13.130796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:13.130351 5139 scope.go:122] "RemoveContainer" containerID="be7974ad0cfe40ea7fac4d396990d64f2b5a1ed5cec82954502dea621a327059" Oct 30 22:19:13.130796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:13.130543 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns" Oct 30 22:19:13.287074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:13.287041 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns"] Oct 30 22:19:13.333597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:13.333568 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-qwsns"] Oct 30 22:19:13.543566 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:13.543537 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" path="/var/lib/kubelet/pods/dd723b38-6cea-4308-bcf3-26369bbfeecc/volumes" Oct 30 22:19:19.813918 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:19:19.813874 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:19:21.976995 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:21.976954 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk"] Oct 30 22:19:21.977637 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:21.977225 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" containerID="cri-o://016abd1ed7520f6bd4932753902f3dfcc61a87b6b14473d84db32e7c3b4a1960" gracePeriod=30 Oct 30 22:19:22.989506 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:22.989472 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:22.989918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:22.989537 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:25.312742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.312706 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr"] Oct 30 22:19:25.313209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313194 5139 cpu_manager.go:414] "containerMap: removing container" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" Oct 30 22:19:25.313253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313209 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" Oct 30 22:19:25.313253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313219 5139 cpu_manager.go:414] "containerMap: removing container" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:25.313253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313224 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:25.313253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313231 5139 cpu_manager.go:414] "containerMap: removing container" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:25.313253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313236 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:25.313253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313246 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ac3b2a5a-8e7d-4131-9354-4aac1f436c08" containerName="maas-api" Oct 30 22:19:25.313253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313251 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3b2a5a-8e7d-4131-9354-4aac1f436c08" containerName="maas-api" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313258 5139 cpu_manager.go:414] "containerMap: removing container" podUID="92b72f34-d257-4fe1-9c9b-6ecab47534bc" containerName="discovery" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313263 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b72f34-d257-4fe1-9c9b-6ecab47534bc" containerName="discovery" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313270 5139 cpu_manager.go:414] "containerMap: removing container" podUID="6c5aec13-d71b-463d-aecc-1ebdffad74b2" containerName="maas-api" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313274 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c5aec13-d71b-463d-aecc-1ebdffad74b2" containerName="maas-api" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313294 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313298 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313308 5139 cpu_manager.go:414] "containerMap: removing container" podUID="a4082918-3ac0-4ca5-aa66-2f3dee8969ab" containerName="maas-api" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313312 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4082918-3ac0-4ca5-aa66-2f3dee8969ab" containerName="maas-api" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313324 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313329 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313340 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313345 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313427 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313435 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313441 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313446 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313451 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313457 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3b2a5a-8e7d-4131-9354-4aac1f436c08" containerName="maas-api" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313463 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b72f34-d257-4fe1-9c9b-6ecab47534bc" containerName="discovery" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313468 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c5aec13-d71b-463d-aecc-1ebdffad74b2" containerName="maas-api" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313476 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313482 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4082918-3ac0-4ca5-aa66-2f3dee8969ab" containerName="maas-api" Oct 30 22:19:25.313498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.313488 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:25.318334 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.318314 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.395107 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.395056 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr"] Oct 30 22:19:25.404718 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.404680 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlfxg\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-kube-api-access-vlfxg\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.404896 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.404758 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-envoy\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.404896 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.404813 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-podinfo\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.404896 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.404845 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda784e-ba36-4d39-9c85-8e0d63656568-istiod-ca-cert\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.405064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.404905 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-socket\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.405064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.404976 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-token\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.405064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.405006 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-credential-socket\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.405326 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.405297 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-data\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.405402 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.405369 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-certs\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.426671 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.426646 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d2f1cf-openshift-default-dockercfg-fnthw\"" Oct 30 22:19:25.506551 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506515 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-token\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.506726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506564 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-credential-socket\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.506726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506626 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-data\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.506726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506670 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-certs\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.506892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506723 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlfxg\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-kube-api-access-vlfxg\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.506892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506773 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-envoy\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.507007 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506883 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-podinfo\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.507007 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506941 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda784e-ba36-4d39-9c85-8e0d63656568-istiod-ca-cert\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.507007 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.506980 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-socket\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.507149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.507037 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-credential-socket\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.507204 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.507164 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-certs\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.507260 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.507240 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-data\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.507489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.507464 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-socket\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.507782 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.507759 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda784e-ba36-4d39-9c85-8e0d63656568-istiod-ca-cert\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.509224 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.509202 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-envoy\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.509534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.509517 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-podinfo\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.564404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.564325 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-token\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.565193 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.565172 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlfxg\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-kube-api-access-vlfxg\") pod \"e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:25.633094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:25.633053 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:26.062344 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:26.062309 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr"] Oct 30 22:19:26.067891 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:19:26.067864 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eda784e_ba36_4d39_9c85_8e0d63656568.slice/crio-92d3e56d9a2f85c3e7c758a9bbc0bc62f3147ffe4cacb4abc085d60e51077be1 WatchSource:0}: Error finding container 92d3e56d9a2f85c3e7c758a9bbc0bc62f3147ffe4cacb4abc085d60e51077be1: Status 404 returned error can't find the container with id 92d3e56d9a2f85c3e7c758a9bbc0bc62f3147ffe4cacb4abc085d60e51077be1 Oct 30 22:19:26.070376 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:26.070341 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:19:26.070462 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:26.070410 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:19:26.070508 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:26.070473 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:19:26.217536 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:26.217498 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" event={"ID":"5eda784e-ba36-4d39-9c85-8e0d63656568","Type":"ContainerStarted","Data":"5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed"} Oct 30 22:19:26.217692 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:26.217571 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" event={"ID":"5eda784e-ba36-4d39-9c85-8e0d63656568","Type":"ContainerStarted","Data":"92d3e56d9a2f85c3e7c758a9bbc0bc62f3147ffe4cacb4abc085d60e51077be1"} Oct 30 22:19:26.333341 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:26.333232 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" podStartSLOduration=1.333214954 podStartE2EDuration="1.333214954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:26.331459616 +0000 UTC m=+2474.922745103" watchObservedRunningTime="2026-10-30 22:19:26.333214954 +0000 UTC m=+2474.924500441" Oct 30 22:19:26.633286 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:26.633210 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:27.226372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.226342 5139 generic.go:356] "Generic (PLEG): container finished" podID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerID="016abd1ed7520f6bd4932753902f3dfcc61a87b6b14473d84db32e7c3b4a1960" exitCode=0 Oct 30 22:19:27.226527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.226406 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" event={"ID":"1cd4804f-6ea9-4033-a077-e9e5a246bcd6","Type":"ContainerDied","Data":"016abd1ed7520f6bd4932753902f3dfcc61a87b6b14473d84db32e7c3b4a1960"} Oct 30 22:19:27.226527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.226480 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" event={"ID":"1cd4804f-6ea9-4033-a077-e9e5a246bcd6","Type":"ContainerDied","Data":"1689e6e715420a83b339424654851639c59dff94b574686c17765e10a1f8361d"} Oct 30 22:19:27.226527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.226501 5139 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1689e6e715420a83b339424654851639c59dff94b574686c17765e10a1f8361d" Oct 30 22:19:27.226527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.226516 5139 scope.go:122] "RemoveContainer" containerID="a33cbea98b2930236368fa8e77612d10e5d3b15b9aa6275bd9968f8391fc3c7c" Oct 30 22:19:27.244399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.244381 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:19:27.330960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.330932 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-socket\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.330960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.330963 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istiod-ca-cert\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.331157 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331016 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-envoy\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.331157 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331050 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-data\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.331157 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331092 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-token\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-token\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.331352 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331179 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-credential-socket\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.331352 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331264 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-kube-api-access-4p2kn\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-kube-api-access-4p2kn\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.331352 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331268 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-socket" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:27.331352 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331315 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-certs\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.331597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331360 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-podinfo\") pod \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\" (UID: \"1cd4804f-6ea9-4033-a077-e9e5a246bcd6\") " Oct 30 22:19:27.331597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331458 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-data" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:27.331597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331536 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-credential-socket" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:27.331597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331574 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istiod-ca-cert" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:27.331771 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.331694 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-certs" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:27.332061 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.332038 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.332182 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.332063 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.332182 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.332079 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.332182 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.332094 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.332182 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.332109 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.333563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.333539 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-kube-api-access-4p2kn" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "kube-api-access-4p2kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:27.333883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.333690 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-podinfo" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:27.333883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.333738 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-token" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:27.333883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.333759 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-envoy" pod "1cd4804f-6ea9-4033-a077-e9e5a246bcd6" (UID: "1cd4804f-6ea9-4033-a077-e9e5a246bcd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:27.433100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.433067 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.433100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.433094 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.433100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.433104 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p2kn\" (UniqueName: \"kubernetes.io/projected/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-kube-api-access-4p2kn\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.433328 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.433113 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1cd4804f-6ea9-4033-a077-e9e5a246bcd6-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:27.639560 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:27.639530 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:28.233273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:28.233243 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk" Oct 30 22:19:28.233680 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:28.233662 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:28.234715 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:28.234692 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:19:28.590220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:28.590182 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk"] Oct 30 22:19:28.597326 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:28.597291 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5d2b13-openshift-default-7879d6b5c5-t9ksk"] Oct 30 22:19:29.543067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:29.543031 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" path="/var/lib/kubelet/pods/1cd4804f-6ea9-4033-a077-e9e5a246bcd6/volumes" Oct 30 22:19:39.062601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:39.062563 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6"] Oct 30 22:19:39.063134 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:39.062933 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" containerID="cri-o://224d7dc9f714aca7ba52569bdaf5ad5880cb2346a424965be0eef8a6bbab86d7" gracePeriod=30 Oct 30 22:19:39.558332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:39.558300 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-6ljr6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:39.558534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:39.558353 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:44.322266 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.322234 5139 generic.go:356] "Generic (PLEG): container finished" podID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerID="224d7dc9f714aca7ba52569bdaf5ad5880cb2346a424965be0eef8a6bbab86d7" exitCode=0 Oct 30 22:19:44.322619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.322316 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" event={"ID":"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6","Type":"ContainerDied","Data":"224d7dc9f714aca7ba52569bdaf5ad5880cb2346a424965be0eef8a6bbab86d7"} Oct 30 22:19:44.322619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.322367 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" event={"ID":"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6","Type":"ContainerDied","Data":"e12a2341eff369691dc04c6866700473c90122a75a5487e41fbb9a9786d3c9e2"} Oct 30 22:19:44.322619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.322385 5139 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e12a2341eff369691dc04c6866700473c90122a75a5487e41fbb9a9786d3c9e2" Oct 30 22:19:44.323233 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.323217 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 22:19:44.463863 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.463771 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-token\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-token\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.463863 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.463854 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-socket\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.464089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.463907 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-certs\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.464089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.463971 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istiod-ca-cert\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.464089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464019 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-data\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.464089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464059 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-envoy\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.464296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464129 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-podinfo\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.464296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464162 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-socket" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:44.464296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464243 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-kube-api-access-sbpn4\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-kube-api-access-sbpn4\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.464296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464251 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-certs" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:44.464530 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464285 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-credential-socket\") pod \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\" (UID: \"00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6\") " Oct 30 22:19:44.464530 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464457 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-data" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:44.464622 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464559 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-credential-socket" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:44.464719 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464696 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istiod-ca-cert" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:44.464885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464863 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.464977 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464903 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.464977 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464921 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.464977 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464934 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.464977 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.464947 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.466656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.466622 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-token" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:44.466748 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.466655 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-podinfo" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:44.466841 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.466823 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-kube-api-access-sbpn4" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "kube-api-access-sbpn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:44.466885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.466822 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-envoy" pod "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" (UID: "00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:44.565711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.565669 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.565711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.565700 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.565711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.565712 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbpn4\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-kube-api-access-sbpn4\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:44.565711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:44.565721 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:45.332004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:45.331970 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6" Oct 30 22:19:45.543486 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:45.543455 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6"] Oct 30 22:19:45.565391 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:45.565358 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-6ljr6"] Oct 30 22:19:47.269651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:47.269608 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2"] Oct 30 22:19:47.270035 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:47.269817 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" containerID="cri-o://c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8" gracePeriod=30 Oct 30 22:19:47.543297 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:47.543224 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" path="/var/lib/kubelet/pods/00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6/volumes" Oct 30 22:19:49.719086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:49.719044 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:19:49.719547 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:49.719110 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:19:49.804935 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:19:49.804893 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:19:52.534159 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.534090 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:19:52.656705 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.656670 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-podinfo\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.656891 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.656794 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-data\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.656891 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.656825 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-kube-api-access-p24zv\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-kube-api-access-p24zv\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.656891 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.656874 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-envoy\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.657064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.656893 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-token\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-token\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.657064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.656934 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-socket\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.657064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.656970 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-certs\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.657064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657005 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istiod-ca-cert\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.657064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657054 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-credential-socket\") pod \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\" (UID: \"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3\") " Oct 30 22:19:52.657333 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657154 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-data" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:52.657333 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657229 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-certs" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:52.657333 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657245 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-socket" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:52.657527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657436 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-credential-socket" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:52.657527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657507 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istiod-ca-cert" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:19:52.657628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657560 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:52.657628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657582 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:52.657628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657594 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:52.657628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.657603 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:52.659137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.659106 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-envoy" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:19:52.659384 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.659360 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-token" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:52.659489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.659448 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-kube-api-access-p24zv" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "kube-api-access-p24zv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:19:52.659489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.659447 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-podinfo" pod "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" (UID: "5f629672-9f9b-41da-b2ff-9bb6fd30d6c3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:19:52.758966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.758929 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p24zv\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-kube-api-access-p24zv\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:52.758966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.758959 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:52.758966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.758969 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:52.759232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.758977 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:52.759232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:52.758986 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:19:53.374953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.374916 5139 generic.go:356] "Generic (PLEG): container finished" podID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerID="c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8" exitCode=0 Oct 30 22:19:53.375239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.374971 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" Oct 30 22:19:53.375239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.375005 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" event={"ID":"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3","Type":"ContainerDied","Data":"c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8"} Oct 30 22:19:53.375239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.375065 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2" event={"ID":"5f629672-9f9b-41da-b2ff-9bb6fd30d6c3","Type":"ContainerDied","Data":"041cc265a418b03f02cb003161a28c5ba175c565fc87b80051e09c981ed4697b"} Oct 30 22:19:53.375239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.375088 5139 scope.go:122] "RemoveContainer" containerID="c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8" Oct 30 22:19:53.387399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.387379 5139 scope.go:122] "RemoveContainer" containerID="fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61" Oct 30 22:19:53.396591 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.396567 5139 scope.go:122] "RemoveContainer" containerID="c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8" Oct 30 22:19:53.396976 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:19:53.396949 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8\": container with ID starting with c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8 not found: ID does not exist" containerID="c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8" Oct 30 22:19:53.397076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.396984 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8"} err="failed to get container status \"c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8\": rpc error: code = NotFound desc = could not find container \"c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8\": container with ID starting with c8cfd8d9fbb1dd7fe4abe9165f5d6b01536ec276fff72dbf90aca04519fd97b8 not found: ID does not exist" Oct 30 22:19:53.397076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.397002 5139 scope.go:122] "RemoveContainer" containerID="fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61" Oct 30 22:19:53.397333 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:19:53.397313 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61\": container with ID starting with fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61 not found: ID does not exist" containerID="fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61" Oct 30 22:19:53.397399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.397345 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61"} err="failed to get container status \"fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61\": rpc error: code = NotFound desc = could not find container \"fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61\": container with ID starting with fc603bd03312dd9de9ab817819e5c7251a47833121d40d07cf6d3fcc7be5ce61 not found: ID does not exist" Oct 30 22:19:53.576316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.576275 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2"] Oct 30 22:19:53.623263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:53.623225 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-fe745d-openshift-default-7979c65b59-p76d2"] Oct 30 22:19:55.543764 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:55.543735 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" path="/var/lib/kubelet/pods/5f629672-9f9b-41da-b2ff-9bb6fd30d6c3/volumes" Oct 30 22:19:58.538099 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538067 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd"] Oct 30 22:19:58.538530 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538516 5139 cpu_manager.go:414] "containerMap: removing container" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" Oct 30 22:19:58.538581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538530 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" Oct 30 22:19:58.538581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538546 5139 cpu_manager.go:414] "containerMap: removing container" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" Oct 30 22:19:58.538581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538551 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" Oct 30 22:19:58.538581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538561 5139 cpu_manager.go:414] "containerMap: removing container" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" Oct 30 22:19:58.538581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538567 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" Oct 30 22:19:58.538581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538573 5139 cpu_manager.go:414] "containerMap: removing container" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" Oct 30 22:19:58.538581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538578 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd723b38-6cea-4308-bcf3-26369bbfeecc" containerName="istio-proxy" Oct 30 22:19:58.538581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538584 5139 cpu_manager.go:414] "containerMap: removing container" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538590 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c3e892-a9e0-4bea-a612-f1ede9d3373e" containerName="cleanup" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538599 5139 cpu_manager.go:414] "containerMap: removing container" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538604 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538612 5139 cpu_manager.go:414] "containerMap: removing container" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538616 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538683 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a114fa-b7ff-4186-ba9e-0a5d0ba29ce6" containerName="istio-proxy" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538690 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538696 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac06a7d6-22f0-4c88-ba1d-4422dfb8aae4" containerName="cleanup" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538703 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" Oct 30 22:19:58.538818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.538710 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd4804f-6ea9-4033-a077-e9e5a246bcd6" containerName="istio-proxy" Oct 30 22:19:58.544050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.544026 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:58.598187 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.598148 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd"] Oct 30 22:19:58.629875 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.629841 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc491e22-ff69-4d08-a8da-40c265bb2d02-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd\" (UID: \"dc491e22-ff69-4d08-a8da-40c265bb2d02\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:58.630050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.629879 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z75xz\" (UniqueName: \"kubernetes.io/projected/dc491e22-ff69-4d08-a8da-40c265bb2d02-kube-api-access-z75xz\") pod \"payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd\" (UID: \"dc491e22-ff69-4d08-a8da-40c265bb2d02\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:58.641285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.641261 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f7c488-dockercfg-x7j94\"" Oct 30 22:19:58.641285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.641264 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f7c488\"" Oct 30 22:19:58.731587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.731550 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc491e22-ff69-4d08-a8da-40c265bb2d02-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd\" (UID: \"dc491e22-ff69-4d08-a8da-40c265bb2d02\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:58.731587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.731587 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z75xz\" (UniqueName: \"kubernetes.io/projected/dc491e22-ff69-4d08-a8da-40c265bb2d02-kube-api-access-z75xz\") pod \"payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd\" (UID: \"dc491e22-ff69-4d08-a8da-40c265bb2d02\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:58.732318 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.732295 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc491e22-ff69-4d08-a8da-40c265bb2d02-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd\" (UID: \"dc491e22-ff69-4d08-a8da-40c265bb2d02\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:58.792239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.792154 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z75xz\" (UniqueName: \"kubernetes.io/projected/dc491e22-ff69-4d08-a8da-40c265bb2d02-kube-api-access-z75xz\") pod \"payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd\" (UID: \"dc491e22-ff69-4d08-a8da-40c265bb2d02\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:58.856510 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:58.856474 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:59.230688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:59.230578 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd"] Oct 30 22:19:59.236391 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:19:59.236364 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc491e22_ff69_4d08_a8da_40c265bb2d02.slice/crio-0b3da50b09bebc287ba1afeba00689a6f6873043b8f119b2f6aacc625743c109 WatchSource:0}: Error finding container 0b3da50b09bebc287ba1afeba00689a6f6873043b8f119b2f6aacc625743c109: Status 404 returned error can't find the container with id 0b3da50b09bebc287ba1afeba00689a6f6873043b8f119b2f6aacc625743c109 Oct 30 22:19:59.412313 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:59.412273 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" event={"ID":"dc491e22-ff69-4d08-a8da-40c265bb2d02","Type":"ContainerStarted","Data":"fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a"} Oct 30 22:19:59.412525 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:59.412333 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" event={"ID":"dc491e22-ff69-4d08-a8da-40c265bb2d02","Type":"ContainerStarted","Data":"0b3da50b09bebc287ba1afeba00689a6f6873043b8f119b2f6aacc625743c109"} Oct 30 22:19:59.412525 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:59.412354 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:19:59.522120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:19:59.522060 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" podStartSLOduration=1.52204579 podStartE2EDuration="1.52204579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:59.520702791 +0000 UTC m=+2508.111988278" watchObservedRunningTime="2026-10-30 22:19:59.52204579 +0000 UTC m=+2508.113331276" Oct 30 22:20:10.421410 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:10.421378 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:20:12.121573 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.121542 5139 scope.go:122] "RemoveContainer" containerID="224d7dc9f714aca7ba52569bdaf5ad5880cb2346a424965be0eef8a6bbab86d7" Oct 30 22:20:12.201208 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.201169 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j"] Oct 30 22:20:12.201728 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.201714 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f629672-9f9b-41da-b2ff-9bb6fd30d6c3" containerName="istio-proxy" Oct 30 22:20:12.206606 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.206582 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.256061 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.256032 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-ca6d0b-serving-cert\"" Oct 30 22:20:12.284466 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.284433 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.284629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.284587 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a200f0-5908-4059-999a-0704935ff375-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.284629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.284622 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.284749 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.284649 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gs2w\" (UniqueName: \"kubernetes.io/projected/58a200f0-5908-4059-999a-0704935ff375-kube-api-access-4gs2w\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.312264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.312229 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j"] Oct 30 22:20:12.385974 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.385881 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a200f0-5908-4059-999a-0704935ff375-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.385974 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.385958 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.386203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.385995 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gs2w\" (UniqueName: \"kubernetes.io/projected/58a200f0-5908-4059-999a-0704935ff375-kube-api-access-4gs2w\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.386203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.386092 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.386334 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:20:12.386311 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-serving-cert: secret "maas-api-e2e-praxis-a-ca6d0b-serving-cert" not found Oct 30 22:20:12.386451 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:20:12.386434 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls podName:58a200f0-5908-4059-999a-0704935ff375 nodeName:}" failed. No retries permitted until 2026-10-30 22:20:12.886386283 +0000 UTC m=+2521.477671770 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls") pod "maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" (UID: "58a200f0-5908-4059-999a-0704935ff375") : secret "maas-api-e2e-praxis-a-ca6d0b-serving-cert" not found Oct 30 22:20:12.386834 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.386806 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a200f0-5908-4059-999a-0704935ff375-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.388589 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.388565 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-metrics-certs\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.443489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.443458 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gs2w\" (UniqueName: \"kubernetes.io/projected/58a200f0-5908-4059-999a-0704935ff375-kube-api-access-4gs2w\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.891007 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.890968 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:12.893430 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:12.893392 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls\") pod \"maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:13.119032 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:13.118994 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:13.492069 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:13.492030 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j"] Oct 30 22:20:13.498866 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:20:13.498831 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a200f0_5908_4059_999a_0704935ff375.slice/crio-90c43b2de1c09c8173557d489b9d3214dd3f3a758437470a84b0fdce439ecda5 WatchSource:0}: Error finding container 90c43b2de1c09c8173557d489b9d3214dd3f3a758437470a84b0fdce439ecda5: Status 404 returned error can't find the container with id 90c43b2de1c09c8173557d489b9d3214dd3f3a758437470a84b0fdce439ecda5 Oct 30 22:20:13.500772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:13.500746 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:20:14.495776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:14.495740 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" event={"ID":"58a200f0-5908-4059-999a-0704935ff375","Type":"ContainerStarted","Data":"90c43b2de1c09c8173557d489b9d3214dd3f3a758437470a84b0fdce439ecda5"} Oct 30 22:20:15.503500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:15.503455 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" event={"ID":"58a200f0-5908-4059-999a-0704935ff375","Type":"ContainerStarted","Data":"10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492"} Oct 30 22:20:15.503923 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:15.503530 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:15.615156 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:15.615098 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" podStartSLOduration=2.621456214 podStartE2EDuration="3.615084914s" totalImagesPullingTime="993.6287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:15.614510843 +0000 UTC m=+2524.205796331" watchObservedRunningTime="2026-10-30 22:20:15.615084914 +0000 UTC m=+2524.206370457" Oct 30 22:20:17.271759 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.271721 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h"] Oct 30 22:20:17.275886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.275862 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.325160 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.325129 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d2f1cf-serving-cert\"" Oct 30 22:20:17.340742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.340711 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.340886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.340798 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwd2z\" (UniqueName: \"kubernetes.io/projected/5d3a9723-dbd9-46bf-be14-2bf7c390030f-kube-api-access-kwd2z\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.340944 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.340911 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-metrics-certs\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.341023 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.341003 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a9723-dbd9-46bf-be14-2bf7c390030f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.387342 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.387305 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h"] Oct 30 22:20:17.442232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.442193 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.442401 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.442271 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwd2z\" (UniqueName: \"kubernetes.io/projected/5d3a9723-dbd9-46bf-be14-2bf7c390030f-kube-api-access-kwd2z\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.442401 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:20:17.442338 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-serving-cert: secret "maas-api-e2e-ipp-b-d2f1cf-serving-cert" not found Oct 30 22:20:17.442591 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.442399 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-metrics-certs\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.442591 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:20:17.442431 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls podName:5d3a9723-dbd9-46bf-be14-2bf7c390030f nodeName:}" failed. No retries permitted until 2026-10-30 22:20:17.942393631 +0000 UTC m=+2526.533679097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls") pod "maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" (UID: "5d3a9723-dbd9-46bf-be14-2bf7c390030f") : secret "maas-api-e2e-ipp-b-d2f1cf-serving-cert" not found Oct 30 22:20:17.442591 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.442554 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a9723-dbd9-46bf-be14-2bf7c390030f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.443212 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.443192 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a9723-dbd9-46bf-be14-2bf7c390030f-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.444804 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.444788 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-metrics-certs\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.500965 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.500925 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwd2z\" (UniqueName: \"kubernetes.io/projected/5d3a9723-dbd9-46bf-be14-2bf7c390030f-kube-api-access-kwd2z\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.948350 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.948304 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:17.950790 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:17.950767 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:18.188749 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:18.188712 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:18.562364 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:18.562326 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h"] Oct 30 22:20:18.568062 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:20:18.568024 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3a9723_dbd9_46bf_be14_2bf7c390030f.slice/crio-d852660e46381a91c07de51622feca29a964fe603bd9aa32183fe89fe28b9a92 WatchSource:0}: Error finding container d852660e46381a91c07de51622feca29a964fe603bd9aa32183fe89fe28b9a92: Status 404 returned error can't find the container with id d852660e46381a91c07de51622feca29a964fe603bd9aa32183fe89fe28b9a92 Oct 30 22:20:19.527305 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:19.527257 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" event={"ID":"5d3a9723-dbd9-46bf-be14-2bf7c390030f","Type":"ContainerStarted","Data":"cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b"} Oct 30 22:20:19.527498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:19.527332 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:19.527498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:19.527351 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" event={"ID":"5d3a9723-dbd9-46bf-be14-2bf7c390030f","Type":"ContainerStarted","Data":"d852660e46381a91c07de51622feca29a964fe603bd9aa32183fe89fe28b9a92"} Oct 30 22:20:19.664177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:19.664074 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" podStartSLOduration=1.848258392 podStartE2EDuration="2.664061684s" totalImagesPullingTime="815.803292ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:19.662690243 +0000 UTC m=+2528.253975729" watchObservedRunningTime="2026-10-30 22:20:19.664061684 +0000 UTC m=+2528.255347162" Oct 30 22:20:19.820647 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:20:19.820603 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:20:21.515552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:21.515466 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:20:25.543389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:25.543360 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:20:49.805139 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:20:49.805090 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:20:50.722888 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:50.722855 5139 generic.go:356] "Generic (PLEG): container finished" podID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerID="5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed" exitCode=0 Oct 30 22:20:50.723058 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:50.722930 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" event={"ID":"5eda784e-ba36-4d39-9c85-8e0d63656568","Type":"ContainerDied","Data":"5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed"} Oct 30 22:20:50.723409 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:50.723394 5139 scope.go:122] "RemoveContainer" containerID="5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed" Oct 30 22:20:50.725010 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:50.724982 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:20:50.725075 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:50.725051 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:20:50.725110 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:50.725087 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:20:51.731235 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:51.731198 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" event={"ID":"5eda784e-ba36-4d39-9c85-8e0d63656568","Type":"ContainerStarted","Data":"ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b"} Oct 30 22:20:52.634169 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:52.634140 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:20:52.634345 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:52.634303 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:20:52.634397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:52.634358 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:20:53.080438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.074279 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj"] Oct 30 22:20:53.091201 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.091173 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.107240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.107202 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.107369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.107352 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.107429 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.107384 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.107429 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.107407 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcnxb\" (UniqueName: \"kubernetes.io/projected/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-kube-api-access-vcnxb\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.132086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.132055 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj"] Oct 30 22:20:53.140125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.140101 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-f7c488-serving-cert\"" Oct 30 22:20:53.208910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.208875 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.209085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.208999 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.209085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.209026 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.209085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.209047 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcnxb\" (UniqueName: \"kubernetes.io/projected/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-kube-api-access-vcnxb\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.209256 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:20:53.209155 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-serving-cert: secret "maas-api-e2e-praxis-a-f7c488-serving-cert" not found Oct 30 22:20:53.209256 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:20:53.209236 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls podName:3184ce2a-e642-4ac6-9bc8-fe1d65caec2f nodeName:}" failed. No retries permitted until 2026-10-30 22:20:53.709212582 +0000 UTC m=+2562.300498059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls") pod "maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" (UID: "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f") : secret "maas-api-e2e-praxis-a-f7c488-serving-cert" not found Oct 30 22:20:53.209805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.209782 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.211447 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.211397 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-metrics-certs\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.289729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.289702 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcnxb\" (UniqueName: \"kubernetes.io/projected/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-kube-api-access-vcnxb\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.634170 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.634141 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:20:53.634346 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.634192 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:20:53.715777 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.715740 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:53.718220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:53.718193 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:54.003651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:54.003613 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:54.390586 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:54.390561 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj"] Oct 30 22:20:54.393399 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:20:54.393368 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3184ce2a_e642_4ac6_9bc8_fe1d65caec2f.slice/crio-99bb1fe079b49d1bf19b8a8ed12719f97d4650e9ca378a29d7ac150ec96e9851 WatchSource:0}: Error finding container 99bb1fe079b49d1bf19b8a8ed12719f97d4650e9ca378a29d7ac150ec96e9851: Status 404 returned error can't find the container with id 99bb1fe079b49d1bf19b8a8ed12719f97d4650e9ca378a29d7ac150ec96e9851 Oct 30 22:20:54.633677 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:54.633640 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:20:54.633854 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:54.633690 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:20:54.750315 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:54.750280 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" event={"ID":"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f","Type":"ContainerStarted","Data":"99bb1fe079b49d1bf19b8a8ed12719f97d4650e9ca378a29d7ac150ec96e9851"} Oct 30 22:20:55.633437 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:55.633386 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:20:55.633850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:55.633442 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:20:55.756934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:55.756900 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" event={"ID":"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f","Type":"ContainerStarted","Data":"43b5496845c699312ab20d6fb6aaa36ec6074006cac36d96c098906c650f121c"} Oct 30 22:20:55.757128 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:55.757028 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:20:55.883440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:55.871115 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" podStartSLOduration=1.868220844 podStartE2EDuration="2.8710949s" totalImagesPullingTime="1.002874056s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:55.869819297 +0000 UTC m=+2564.461104784" watchObservedRunningTime="2026-10-30 22:20:55.8710949 +0000 UTC m=+2564.462380392" Oct 30 22:20:56.633840 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:56.633802 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body= Oct 30 22:20:56.634307 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:56.633854 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" Oct 30 22:20:57.640446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:57.640388 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:20:57.640943 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:57.640843 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:20:57.641651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:20:57.641631 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:21:01.769903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:01.769871 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:21:09.723721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:09.723687 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh"] Oct 30 22:21:09.724225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:09.723969 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" containerID="cri-o://5ca8f63063850253d74a219f9a42a021dc9392397d3b15002b4a8999c78aa422" gracePeriod=30 Oct 30 22:21:12.388087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:12.388053 5139 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-pwbsh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:21:12.388481 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:12.388116 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:21:14.860323 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:14.860286 5139 generic.go:356] "Generic (PLEG): container finished" podID="273029a7-c256-495f-8a15-8b8a295e0885" containerID="5ca8f63063850253d74a219f9a42a021dc9392397d3b15002b4a8999c78aa422" exitCode=0 Oct 30 22:21:14.860817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:14.860341 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" event={"ID":"273029a7-c256-495f-8a15-8b8a295e0885","Type":"ContainerDied","Data":"5ca8f63063850253d74a219f9a42a021dc9392397d3b15002b4a8999c78aa422"} Oct 30 22:21:14.983814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:14.983789 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 22:21:15.157315 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157207 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-data\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.157494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157346 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-envoy\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.157494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157391 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-istio-token\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-istio-token\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.157494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157446 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/273029a7-c256-495f-8a15-8b8a295e0885-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273029a7-c256-495f-8a15-8b8a295e0885-istio-podinfo\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.157665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157509 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273029a7-c256-495f-8a15-8b8a295e0885-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273029a7-c256-495f-8a15-8b8a295e0885-istiod-ca-cert\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.157665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157535 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-data" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:15.157665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157555 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-credential-socket\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.157665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157589 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-certs\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.157665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157621 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-socket\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.157665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.157656 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-kube-api-access-2lxkl\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-kube-api-access-2lxkl\") pod \"273029a7-c256-495f-8a15-8b8a295e0885\" (UID: \"273029a7-c256-495f-8a15-8b8a295e0885\") " Oct 30 22:21:15.158475 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.158151 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-certs" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:15.158475 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.158168 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.158742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.158588 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-credential-socket" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:15.158928 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.158906 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-socket" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:15.159249 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.159229 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273029a7-c256-495f-8a15-8b8a295e0885-istiod-ca-cert" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:21:15.160993 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.160951 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-kube-api-access-2lxkl" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "kube-api-access-2lxkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:15.160993 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.160952 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/273029a7-c256-495f-8a15-8b8a295e0885-istio-podinfo" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:21:15.161144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.161003 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-istio-token" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:21:15.168806 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.168779 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-envoy" pod "273029a7-c256-495f-8a15-8b8a295e0885" (UID: "273029a7-c256-495f-8a15-8b8a295e0885"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:21:15.259029 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.258999 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.259029 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.259027 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.259241 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.259037 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.259241 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.259070 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lxkl\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-kube-api-access-2lxkl\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.259241 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.259080 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/273029a7-c256-495f-8a15-8b8a295e0885-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.259241 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.259088 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/273029a7-c256-495f-8a15-8b8a295e0885-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.259241 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.259096 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/273029a7-c256-495f-8a15-8b8a295e0885-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.259241 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.259105 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/273029a7-c256-495f-8a15-8b8a295e0885-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:21:15.867189 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.867103 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" Oct 30 22:21:15.867637 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.867095 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh" event={"ID":"273029a7-c256-495f-8a15-8b8a295e0885","Type":"ContainerDied","Data":"cd97e5fd7b926f0899d7c0fe0241fb5090a2b7b3c48cf8be36ab8ed78ba3116e"} Oct 30 22:21:15.867637 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:15.867244 5139 scope.go:122] "RemoveContainer" containerID="5ca8f63063850253d74a219f9a42a021dc9392397d3b15002b4a8999c78aa422" Oct 30 22:21:16.031970 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:16.031936 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh"] Oct 30 22:21:16.078211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:16.078174 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-pwbsh"] Oct 30 22:21:17.543240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:17.543196 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273029a7-c256-495f-8a15-8b8a295e0885" path="/var/lib/kubelet/pods/273029a7-c256-495f-8a15-8b8a295e0885/volumes" Oct 30 22:21:19.804882 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:21:19.804837 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:21:21.347958 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.347926 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k"] Oct 30 22:21:21.348401 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.348387 5139 cpu_manager.go:414] "containerMap: removing container" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" Oct 30 22:21:21.348482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.348401 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" Oct 30 22:21:21.348518 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.348507 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="273029a7-c256-495f-8a15-8b8a295e0885" containerName="istio-proxy" Oct 30 22:21:21.352048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.352030 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.401364 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.401334 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f7c488-openshift-default-dockercfg-2kvn7\"" Oct 30 22:21:21.433519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433490 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx6v6\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-kube-api-access-hx6v6\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.433617 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433529 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-podinfo\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.433617 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433605 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.433707 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433638 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9685434f-14f2-4ab9-b01a-e90458d910e3-istiod-ca-cert\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.433755 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433739 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-token\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.433800 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433771 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-credential-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.433837 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433801 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-envoy\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.433889 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433875 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-data\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.433925 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.433902 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-certs\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.459676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.459645 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k"] Oct 30 22:21:21.534951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.534920 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx6v6\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-kube-api-access-hx6v6\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.534964 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-podinfo\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535021 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535046 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9685434f-14f2-4ab9-b01a-e90458d910e3-istiod-ca-cert\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535103 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-token\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535461 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-credential-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535705 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-envoy\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535546 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535757 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-credential-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535802 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-data\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535848 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-certs\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.535973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.535912 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9685434f-14f2-4ab9-b01a-e90458d910e3-istiod-ca-cert\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.536344 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.536218 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-data\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.536387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.536345 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-certs\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.538048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.538015 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-envoy\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.538172 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.538155 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-podinfo\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.593086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.593048 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-token\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.593226 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.593169 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx6v6\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-kube-api-access-hx6v6\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:21.663761 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:21.663686 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:22.089939 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:22.089904 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k"] Oct 30 22:21:22.097835 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:21:22.097805 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9685434f_14f2_4ab9_b01a_e90458d910e3.slice/crio-c06c9d8be1955bd7fea0ba0d5739179750ceedc347ec1e09f9fccd649b82ce7e WatchSource:0}: Error finding container c06c9d8be1955bd7fea0ba0d5739179750ceedc347ec1e09f9fccd649b82ce7e: Status 404 returned error can't find the container with id c06c9d8be1955bd7fea0ba0d5739179750ceedc347ec1e09f9fccd649b82ce7e Oct 30 22:21:22.100396 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:22.100367 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:22.100494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:22.100460 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:22.100540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:22.100492 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:22.906486 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:22.906451 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" event={"ID":"9685434f-14f2-4ab9-b01a-e90458d910e3","Type":"ContainerStarted","Data":"e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a"} Oct 30 22:21:22.906885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:22.906507 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" event={"ID":"9685434f-14f2-4ab9-b01a-e90458d910e3","Type":"ContainerStarted","Data":"c06c9d8be1955bd7fea0ba0d5739179750ceedc347ec1e09f9fccd649b82ce7e"} Oct 30 22:21:23.679193 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:23.665719 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:23.686436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:23.682539 5139 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:21:23.686436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:23.682610 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:21:24.686440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:24.684845 5139 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:21:24.686440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:24.684926 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:21:25.684522 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.684481 5139 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:21:25.684783 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.684545 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:21:25.701287 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.701238 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" podStartSLOduration=4.701224491 podStartE2EDuration="4.701224491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.090387267 +0000 UTC m=+2591.681672755" watchObservedRunningTime="2026-10-30 22:21:25.701224491 +0000 UTC m=+2594.292509978" Oct 30 22:21:25.703136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.703111 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx"] Oct 30 22:21:25.706198 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.706179 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.755738 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.755708 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-fc3918-openshift-default-dockercfg-qqb4n\"" Oct 30 22:21:25.808297 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.808257 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-credential-socket\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.808523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.808457 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-token\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.808620 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.808518 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d30451-0c2e-424d-a84e-2085794abd7d-istiod-ca-cert\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.808686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.808627 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-socket\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.808686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.808679 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-podinfo\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.808822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.808797 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-envoy\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.808884 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.808874 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmnc\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-kube-api-access-5hmnc\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.808953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.808908 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-data\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.809163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.809141 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-certs\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.820156 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.820123 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx"] Oct 30 22:21:25.910133 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910045 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-socket\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910168 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-podinfo\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910200 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-envoy\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910262 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmnc\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-kube-api-access-5hmnc\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910288 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-data\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910307 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-certs\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910383 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-credential-socket\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910389 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-socket\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910710 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910589 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-token\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910710 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910630 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-data\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910710 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910652 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d30451-0c2e-424d-a84e-2085794abd7d-istiod-ca-cert\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910837 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910714 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-certs\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.910837 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.910804 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-credential-socket\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.911276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.911259 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d30451-0c2e-424d-a84e-2085794abd7d-istiod-ca-cert\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.912631 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.912608 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-envoy\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.912716 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.912674 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-podinfo\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.991875 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.991838 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-token\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:25.992068 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:25.991988 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmnc\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-kube-api-access-5hmnc\") pod \"e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:26.019070 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.019042 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:26.480819 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.480672 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx"] Oct 30 22:21:26.489131 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:21:26.489094 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9d30451_0c2e_424d_a84e_2085794abd7d.slice/crio-a7e2f3bffac19e7018a342a8941d2d3854d0044824774611392a8adc931a3f89 WatchSource:0}: Error finding container a7e2f3bffac19e7018a342a8941d2d3854d0044824774611392a8adc931a3f89: Status 404 returned error can't find the container with id a7e2f3bffac19e7018a342a8941d2d3854d0044824774611392a8adc931a3f89 Oct 30 22:21:26.492227 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.492187 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:26.492333 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.492288 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:26.492333 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.492326 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:21:26.691440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.689854 5139 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:21:26.691440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.689914 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:21:26.932478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.932433 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" event={"ID":"f9d30451-0c2e-424d-a84e-2085794abd7d","Type":"ContainerStarted","Data":"6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9"} Oct 30 22:21:26.932865 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:26.932500 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" event={"ID":"f9d30451-0c2e-424d-a84e-2085794abd7d","Type":"ContainerStarted","Data":"a7e2f3bffac19e7018a342a8941d2d3854d0044824774611392a8adc931a3f89"} Oct 30 22:21:27.019374 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:27.019338 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:27.021023 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:27.020995 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:21:27.021142 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:27.021045 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:21:27.050280 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:27.050231 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podStartSLOduration=2.050216288 podStartE2EDuration="2.050216288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:27.047917097 +0000 UTC m=+2595.639202587" watchObservedRunningTime="2026-10-30 22:21:27.050216288 +0000 UTC m=+2595.641501775" Oct 30 22:21:27.687603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:27.684027 5139 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 22:21:27.687603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:27.684103 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 22:21:28.024487 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:28.024457 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:28.668988 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:28.668954 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:28.669243 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:28.669223 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:28.670031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:28.670013 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:21:28.943368 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:28.943276 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:28.944351 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:28.944330 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:21:49.804843 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:21:49.804807 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:21:51.409940 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.409862 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc"] Oct 30 22:21:51.414089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.414066 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:51.463552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.463518 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-93131d\"" Oct 30 22:21:51.513305 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.513272 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc"] Oct 30 22:21:51.605290 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.605253 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmd5\" (UniqueName: \"kubernetes.io/projected/9cdd1485-7c01-476b-97b9-6155d59a1a76-kube-api-access-hvmd5\") pod \"payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc\" (UID: \"9cdd1485-7c01-476b-97b9-6155d59a1a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:51.605467 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.605320 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd1485-7c01-476b-97b9-6155d59a1a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc\" (UID: \"9cdd1485-7c01-476b-97b9-6155d59a1a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:51.706700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.706626 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmd5\" (UniqueName: \"kubernetes.io/projected/9cdd1485-7c01-476b-97b9-6155d59a1a76-kube-api-access-hvmd5\") pod \"payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc\" (UID: \"9cdd1485-7c01-476b-97b9-6155d59a1a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:51.706700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.706677 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd1485-7c01-476b-97b9-6155d59a1a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc\" (UID: \"9cdd1485-7c01-476b-97b9-6155d59a1a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:51.707364 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.707345 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd1485-7c01-476b-97b9-6155d59a1a76-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc\" (UID: \"9cdd1485-7c01-476b-97b9-6155d59a1a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:51.765004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:51.764968 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmd5\" (UniqueName: \"kubernetes.io/projected/9cdd1485-7c01-476b-97b9-6155d59a1a76-kube-api-access-hvmd5\") pod \"payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc\" (UID: \"9cdd1485-7c01-476b-97b9-6155d59a1a76\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:52.027325 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:52.027291 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:52.399574 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:52.399491 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc"] Oct 30 22:21:52.406287 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:21:52.406249 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdd1485_7c01_476b_97b9_6155d59a1a76.slice/crio-9d31fc6bd67e485480d7db7a71ae76604f14c799396f3e212942079705f8416d WatchSource:0}: Error finding container 9d31fc6bd67e485480d7db7a71ae76604f14c799396f3e212942079705f8416d: Status 404 returned error can't find the container with id 9d31fc6bd67e485480d7db7a71ae76604f14c799396f3e212942079705f8416d Oct 30 22:21:53.081377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:53.081337 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" event={"ID":"9cdd1485-7c01-476b-97b9-6155d59a1a76","Type":"ContainerStarted","Data":"974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3"} Oct 30 22:21:53.081792 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:53.081386 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" event={"ID":"9cdd1485-7c01-476b-97b9-6155d59a1a76","Type":"ContainerStarted","Data":"9d31fc6bd67e485480d7db7a71ae76604f14c799396f3e212942079705f8416d"} Oct 30 22:21:53.081792 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:53.081453 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:21:53.194141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:53.194098 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" podStartSLOduration=2.194085704 podStartE2EDuration="2.194085704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:53.193627268 +0000 UTC m=+2621.784912758" watchObservedRunningTime="2026-10-30 22:21:53.194085704 +0000 UTC m=+2621.785371190" Oct 30 22:21:59.066848 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.066812 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt"] Oct 30 22:21:59.072723 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.072701 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.122046 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.122024 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-85d00c-serving-cert\"" Oct 30 22:21:59.171994 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.171965 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt"] Oct 30 22:21:59.195671 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.195637 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.195810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.195761 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.195872 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.195814 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-metrics-certs\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.195910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.195892 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgq8k\" (UniqueName: \"kubernetes.io/projected/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-kube-api-access-cgq8k\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.296897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.296862 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.297040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.296907 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-metrics-certs\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.297040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.296941 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgq8k\" (UniqueName: \"kubernetes.io/projected/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-kube-api-access-cgq8k\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.297125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.297070 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.297250 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:21:59.297230 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-serving-cert: secret "maas-api-e2e-praxis-b-85d00c-serving-cert" not found Oct 30 22:21:59.297331 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:21:59.297318 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls podName:2713cb58-da4f-430e-97f4-4c3cd1d5bf0d nodeName:}" failed. No retries permitted until 2026-10-30 22:21:59.797298032 +0000 UTC m=+2628.388583512 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls") pod "maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" (UID: "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d") : secret "maas-api-e2e-praxis-b-85d00c-serving-cert" not found Oct 30 22:21:59.297657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.297640 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.299452 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.299431 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-metrics-certs\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.355866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.355777 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgq8k\" (UniqueName: \"kubernetes.io/projected/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-kube-api-access-cgq8k\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.801397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.801368 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.803874 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.803844 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls\") pod \"maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:21:59.985920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:21:59.985890 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:22:00.357377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:00.357346 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt"] Oct 30 22:22:00.363949 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:22:00.363924 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2713cb58_da4f_430e_97f4_4c3cd1d5bf0d.slice/crio-af5040add0ff786fc5724070aa3005c207d46cbb20d70ae75a133681620610ef WatchSource:0}: Error finding container af5040add0ff786fc5724070aa3005c207d46cbb20d70ae75a133681620610ef: Status 404 returned error can't find the container with id af5040add0ff786fc5724070aa3005c207d46cbb20d70ae75a133681620610ef Oct 30 22:22:01.126022 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:01.125985 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" event={"ID":"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d","Type":"ContainerStarted","Data":"af5040add0ff786fc5724070aa3005c207d46cbb20d70ae75a133681620610ef"} Oct 30 22:22:02.133207 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:02.133169 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" event={"ID":"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d","Type":"ContainerStarted","Data":"339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29"} Oct 30 22:22:02.133692 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:02.133282 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:22:02.288779 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:02.288714 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" podStartSLOduration=2.2356861439999998 podStartE2EDuration="3.288699203s" totalImagesPullingTime="1.053013059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:02.287893357 +0000 UTC m=+2630.879178843" watchObservedRunningTime="2026-10-30 22:22:02.288699203 +0000 UTC m=+2630.879984689" Oct 30 22:22:04.090506 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:04.090477 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:22:08.149306 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:08.149277 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:22:12.163209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:12.163177 5139 scope.go:122] "RemoveContainer" containerID="016abd1ed7520f6bd4932753902f3dfcc61a87b6b14473d84db32e7c3b4a1960" Oct 30 22:22:19.804346 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:22:19.804309 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:22:35.956900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:35.956867 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b"] Oct 30 22:22:35.962429 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:35.962398 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.011466 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.011442 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-fc3918-serving-cert\"" Oct 30 22:22:36.060106 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.060074 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b"] Oct 30 22:22:36.108040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.107997 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.108215 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.108076 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lzkr\" (UniqueName: \"kubernetes.io/projected/1423d7c3-8429-454a-a50e-b85602bb92d8-kube-api-access-6lzkr\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.108215 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.108191 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-metrics-certs\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.108347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.108269 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1423d7c3-8429-454a-a50e-b85602bb92d8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.210205 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.210110 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.210387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.210208 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lzkr\" (UniqueName: \"kubernetes.io/projected/1423d7c3-8429-454a-a50e-b85602bb92d8-kube-api-access-6lzkr\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.210387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.210268 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-metrics-certs\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.210387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.210314 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1423d7c3-8429-454a-a50e-b85602bb92d8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.211015 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.210989 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1423d7c3-8429-454a-a50e-b85602bb92d8-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.212785 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.212759 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-maas-api-tls\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.212890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.212865 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-metrics-certs\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.268395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.268366 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lzkr\" (UniqueName: \"kubernetes.io/projected/1423d7c3-8429-454a-a50e-b85602bb92d8-kube-api-access-6lzkr\") pod \"maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.275250 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.275227 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:36.554353 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.554322 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m"] Oct 30 22:22:36.559527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.559503 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.616395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.616369 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-85d00c-openshift-default-dockercfg-tsjwc\"" Oct 30 22:22:36.643983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.643955 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m"] Oct 30 22:22:36.672728 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.672699 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b"] Oct 30 22:22:36.678882 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:22:36.678855 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1423d7c3_8429_454a_a50e_b85602bb92d8.slice/crio-eda159042ac17ad48e89769ef08afe3cf21142a3576569a34f6f30e8e1aeff81 WatchSource:0}: Error finding container eda159042ac17ad48e89769ef08afe3cf21142a3576569a34f6f30e8e1aeff81: Status 404 returned error can't find the container with id eda159042ac17ad48e89769ef08afe3cf21142a3576569a34f6f30e8e1aeff81 Oct 30 22:22:36.716151 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716115 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-istio-token\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.716322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716165 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8721f2-f926-4225-916b-a37cd5925fda-istio-podinfo\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.716386 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716308 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj7js\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-kube-api-access-qj7js\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.716386 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716366 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.716541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716521 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8721f2-f926-4225-916b-a37cd5925fda-istiod-ca-cert\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.716632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716570 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-certs\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.716859 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716838 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-envoy\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.716921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716884 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-data\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.716960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.716922 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-credential-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.818652 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.818567 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-istio-token\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.818652 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.818602 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8721f2-f926-4225-916b-a37cd5925fda-istio-podinfo\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.818652 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.818629 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj7js\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-kube-api-access-qj7js\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.818932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.818739 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.818932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.818794 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8721f2-f926-4225-916b-a37cd5925fda-istiod-ca-cert\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.818932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.818826 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-certs\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.819082 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.819012 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-envoy\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.819082 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.819054 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-data\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.819184 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.819085 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-credential-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.819184 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.819137 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.819306 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.819254 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-certs\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.819480 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.819451 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-data\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.819617 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.819537 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-credential-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.819617 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.819584 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8721f2-f926-4225-916b-a37cd5925fda-istiod-ca-cert\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.821065 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.821048 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-envoy\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.821183 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.821167 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8721f2-f926-4225-916b-a37cd5925fda-istio-podinfo\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.876711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.876685 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-istio-token\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:36.876934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:36.876915 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj7js\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-kube-api-access-qj7js\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:37.175189 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:37.175110 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:37.340822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:37.340785 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" event={"ID":"1423d7c3-8429-454a-a50e-b85602bb92d8","Type":"ContainerStarted","Data":"eda159042ac17ad48e89769ef08afe3cf21142a3576569a34f6f30e8e1aeff81"} Oct 30 22:22:37.606437 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:37.606393 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m"] Oct 30 22:22:37.612176 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:22:37.612147 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b8721f2_f926_4225_916b_a37cd5925fda.slice/crio-04c3d23b4d7b910f5e4de8a669517e93c73a63f2a865b7598c9484f8f291dda8 WatchSource:0}: Error finding container 04c3d23b4d7b910f5e4de8a669517e93c73a63f2a865b7598c9484f8f291dda8: Status 404 returned error can't find the container with id 04c3d23b4d7b910f5e4de8a669517e93c73a63f2a865b7598c9484f8f291dda8 Oct 30 22:22:37.615496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:37.615456 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:22:37.615581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:37.615552 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:22:37.615624 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:37.615596 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:22:38.348793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:38.348743 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" event={"ID":"7b8721f2-f926-4225-916b-a37cd5925fda","Type":"ContainerStarted","Data":"6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb"} Oct 30 22:22:38.349247 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:38.348857 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" event={"ID":"7b8721f2-f926-4225-916b-a37cd5925fda","Type":"ContainerStarted","Data":"04c3d23b4d7b910f5e4de8a669517e93c73a63f2a865b7598c9484f8f291dda8"} Oct 30 22:22:38.350586 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:38.350552 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" event={"ID":"1423d7c3-8429-454a-a50e-b85602bb92d8","Type":"ContainerStarted","Data":"170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348"} Oct 30 22:22:38.350699 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:38.350641 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:38.560065 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:38.560014 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" podStartSLOduration=2.560001002 podStartE2EDuration="2.560001002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:38.512247072 +0000 UTC m=+2667.103532560" watchObservedRunningTime="2026-10-30 22:22:38.560001002 +0000 UTC m=+2667.151286522" Oct 30 22:22:38.624316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:38.624219 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" podStartSLOduration=2.755168797 podStartE2EDuration="3.62420671s" totalImagesPullingTime="869.037913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:38.622281554 +0000 UTC m=+2667.213567040" watchObservedRunningTime="2026-10-30 22:22:38.62420671 +0000 UTC m=+2667.215492233" Oct 30 22:22:39.187503 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:39.183156 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:39.187503 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:39.183298 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:22:39.187503 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:39.183361 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 22:22:40.183211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:40.182757 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:22:40.183211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:40.182829 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 22:22:41.184439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:41.183478 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:22:41.184439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:41.183540 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 22:22:42.182439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:42.181820 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:22:42.182439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:42.181883 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 22:22:43.184440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:43.182729 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 30 22:22:43.184440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:43.182788 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 30 22:22:44.180802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:44.180770 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:44.181202 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:44.181181 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:44.181843 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:44.181826 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:22:44.361618 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:22:44.361590 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:22:49.805183 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:22:49.805149 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:23:13.569586 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:13.569553 5139 generic.go:356] "Generic (PLEG): container finished" podID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerID="6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9" exitCode=0 Oct 30 22:23:13.569586 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:13.569591 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" event={"ID":"f9d30451-0c2e-424d-a84e-2085794abd7d","Type":"ContainerDied","Data":"6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9"} Oct 30 22:23:13.570042 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:13.569914 5139 scope.go:122] "RemoveContainer" containerID="6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9" Oct 30 22:23:13.571378 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:13.571348 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:13.571462 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:13.571447 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:13.571509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:13.571485 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:23:13.943326 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:13.943237 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:23:14.595891 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:14.595846 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" event={"ID":"f9d30451-0c2e-424d-a84e-2085794abd7d","Type":"ContainerStarted","Data":"a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed"} Oct 30 22:23:15.019859 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:15.019825 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:23:15.020853 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:15.020829 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:23:15.020970 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:15.020882 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:23:16.019588 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:16.019549 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:23:16.019996 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:16.019613 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:23:17.019848 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:17.019815 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:23:17.020245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:17.019878 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:23:18.019521 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:18.019484 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:23:18.019702 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:18.019558 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:23:19.019692 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:19.019655 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 22:23:19.020119 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:19.019718 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 22:23:19.806869 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:23:19.806827 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:23:20.024194 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:20.024165 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:23:20.024691 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:20.024584 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:23:20.025264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:23:20.025243 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:23:49.805790 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:23:49.805748 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:24:19.804599 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:24:19.804562 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:24:29.131246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.131213 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt"] Oct 30 22:24:29.131838 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.131399 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" podUID="2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" containerName="maas-api" containerID="cri-o://339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29" gracePeriod=30 Oct 30 22:24:29.404245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.404222 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:24:29.568312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.568276 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls\") pod \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " Oct 30 22:24:29.568527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.568324 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-kube-api-access-cgq8k\" (UniqueName: \"kubernetes.io/projected/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-kube-api-access-cgq8k\") pod \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " Oct 30 22:24:29.568611 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.568589 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-metrics-certs\") pod \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " Oct 30 22:24:29.568823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.568804 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-openshift-service-ca\") pod \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\" (UID: \"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d\") " Oct 30 22:24:29.569309 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.569280 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-openshift-service-ca" pod "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" (UID: "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:29.569676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.569647 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.570800 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.570777 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls" pod "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" (UID: "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:29.570920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.570872 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-metrics-certs" pod "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" (UID: "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:29.570920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.570899 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-kube-api-access-cgq8k" pod "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" (UID: "2713cb58-da4f-430e-97f4-4c3cd1d5bf0d"). InnerVolumeSpecName "kube-api-access-cgq8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:29.618657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.618632 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ccz7d_9bbdcb9f-b9c8-46b3-8925-7c246f2affe3/payload-processing/0.log" Oct 30 22:24:29.671067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.670993 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.671270 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.671241 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgq8k\" (UniqueName: \"kubernetes.io/projected/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-kube-api-access-cgq8k\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.671397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.671275 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:29.880582 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:29.880554 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ccz7d_9bbdcb9f-b9c8-46b3-8925-7c246f2affe3/payload-processing/0.log" Oct 30 22:24:30.019089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.019055 5139 generic.go:356] "Generic (PLEG): container finished" podID="2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" containerID="339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29" exitCode=0 Oct 30 22:24:30.019275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.019112 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" Oct 30 22:24:30.019275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.019125 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" event={"ID":"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d","Type":"ContainerDied","Data":"339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29"} Oct 30 22:24:30.019275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.019183 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt" event={"ID":"2713cb58-da4f-430e-97f4-4c3cd1d5bf0d","Type":"ContainerDied","Data":"af5040add0ff786fc5724070aa3005c207d46cbb20d70ae75a133681620610ef"} Oct 30 22:24:30.019275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.019214 5139 scope.go:122] "RemoveContainer" containerID="339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29" Oct 30 22:24:30.029566 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.029545 5139 scope.go:122] "RemoveContainer" containerID="339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29" Oct 30 22:24:30.029843 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:24:30.029821 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29\": container with ID starting with 339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29 not found: ID does not exist" containerID="339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29" Oct 30 22:24:30.029936 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.029849 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29"} err="failed to get container status \"339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29\": rpc error: code = NotFound desc = could not find container \"339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29\": container with ID starting with 339b96f9123c29226cd1bd9dd2b12f7450a1c04c5b4382f8de7aa6e695c54a29 not found: ID does not exist" Oct 30 22:24:30.175527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.175496 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt"] Oct 30 22:24:30.180469 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:30.180436 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-85d00c-564f7f8599-p8vzt"] Oct 30 22:24:31.551820 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:31.551786 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" path="/var/lib/kubelet/pods/2713cb58-da4f-430e-97f4-4c3cd1d5bf0d/volumes" Oct 30 22:24:41.665021 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:41.664987 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m"] Oct 30 22:24:41.665612 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:41.665193 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" containerID="cri-o://6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb" gracePeriod=30 Oct 30 22:24:44.183256 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:44.183217 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:24:44.183763 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:44.183269 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:24:46.934964 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:46.934900 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:24:47.054334 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054304 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-socket\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.054505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054354 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b8721f2-f926-4225-916b-a37cd5925fda-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8721f2-f926-4225-916b-a37cd5925fda-istio-podinfo\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.054505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054380 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-data\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.054505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054403 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-credential-socket\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.054505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054446 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-istio-token\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-istio-token\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.054505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054473 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-certs\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.054794 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054512 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b8721f2-f926-4225-916b-a37cd5925fda-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8721f2-f926-4225-916b-a37cd5925fda-istiod-ca-cert\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.054794 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054626 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-socket" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.054794 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054636 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-data" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.054794 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054643 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-envoy\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.054794 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054725 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-kube-api-access-qj7js\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-kube-api-access-qj7js\") pod \"7b8721f2-f926-4225-916b-a37cd5925fda\" (UID: \"7b8721f2-f926-4225-916b-a37cd5925fda\") " Oct 30 22:24:47.055079 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054804 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-credential-socket" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.055079 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054890 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-certs" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.055079 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.054965 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b8721f2-f926-4225-916b-a37cd5925fda-istiod-ca-cert" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:47.055238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.055222 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.055298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.055247 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.055298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.055263 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.055298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.055278 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.055298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.055289 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b8721f2-f926-4225-916b-a37cd5925fda-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.056719 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.056689 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b8721f2-f926-4225-916b-a37cd5925fda-istio-podinfo" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:24:47.056868 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.056852 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-kube-api-access-qj7js" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "kube-api-access-qj7js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:47.057097 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.057081 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-envoy" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:24:47.057160 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.057113 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-istio-token" pod "7b8721f2-f926-4225-916b-a37cd5925fda" (UID: "7b8721f2-f926-4225-916b-a37cd5925fda"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:47.112125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.112080 5139 generic.go:356] "Generic (PLEG): container finished" podID="7b8721f2-f926-4225-916b-a37cd5925fda" containerID="6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb" exitCode=0 Oct 30 22:24:47.112258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.112146 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" Oct 30 22:24:47.112258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.112151 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" event={"ID":"7b8721f2-f926-4225-916b-a37cd5925fda","Type":"ContainerDied","Data":"6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb"} Oct 30 22:24:47.112258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.112208 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m" event={"ID":"7b8721f2-f926-4225-916b-a37cd5925fda","Type":"ContainerDied","Data":"04c3d23b4d7b910f5e4de8a669517e93c73a63f2a865b7598c9484f8f291dda8"} Oct 30 22:24:47.112258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.112235 5139 scope.go:122] "RemoveContainer" containerID="6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb" Oct 30 22:24:47.123865 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.123844 5139 scope.go:122] "RemoveContainer" containerID="6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb" Oct 30 22:24:47.124164 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:24:47.124141 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb\": container with ID starting with 6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb not found: ID does not exist" containerID="6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb" Oct 30 22:24:47.124267 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.124171 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb"} err="failed to get container status \"6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb\": rpc error: code = NotFound desc = could not find container \"6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb\": container with ID starting with 6c611b79a02b685bc918f2f236ef5a7732fca5153b06cdb3dcf4d50ed33e03bb not found: ID does not exist" Oct 30 22:24:47.156283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.156255 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b8721f2-f926-4225-916b-a37cd5925fda-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.156387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.156286 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj7js\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-kube-api-access-qj7js\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.156387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.156301 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b8721f2-f926-4225-916b-a37cd5925fda-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.156387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.156313 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b8721f2-f926-4225-916b-a37cd5925fda-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:47.275478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.275448 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m"] Oct 30 22:24:47.277273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.277247 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-zft7m"] Oct 30 22:24:47.542526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:47.542445 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" path="/var/lib/kubelet/pods/7b8721f2-f926-4225-916b-a37cd5925fda/volumes" Oct 30 22:24:49.805451 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:24:49.805403 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:24:51.362624 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.362551 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j"] Oct 30 22:24:51.363020 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.362774 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" podUID="58a200f0-5908-4059-999a-0704935ff375" containerName="maas-api" containerID="cri-o://10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492" gracePeriod=30 Oct 30 22:24:51.509823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.509778 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" podUID="58a200f0-5908-4059-999a-0704935ff375" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.95:8443/health\": dial tcp 10.134.0.95:8443: connect: connection refused" Oct 30 22:24:51.642864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.642841 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:24:51.816990 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.816957 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a200f0-5908-4059-999a-0704935ff375-kube-api-access-4gs2w\" (UniqueName: \"kubernetes.io/projected/58a200f0-5908-4059-999a-0704935ff375-kube-api-access-4gs2w\") pod \"58a200f0-5908-4059-999a-0704935ff375\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " Oct 30 22:24:51.817168 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.817087 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls\") pod \"58a200f0-5908-4059-999a-0704935ff375\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " Oct 30 22:24:51.817220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.817166 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a200f0-5908-4059-999a-0704935ff375-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a200f0-5908-4059-999a-0704935ff375-openshift-service-ca\") pod \"58a200f0-5908-4059-999a-0704935ff375\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " Oct 30 22:24:51.817220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.817212 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-metrics-certs\") pod \"58a200f0-5908-4059-999a-0704935ff375\" (UID: \"58a200f0-5908-4059-999a-0704935ff375\") " Oct 30 22:24:51.817598 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.817566 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a200f0-5908-4059-999a-0704935ff375-openshift-service-ca" pod "58a200f0-5908-4059-999a-0704935ff375" (UID: "58a200f0-5908-4059-999a-0704935ff375"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:24:51.817730 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.817683 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a200f0-5908-4059-999a-0704935ff375-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:51.819149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.819125 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a200f0-5908-4059-999a-0704935ff375-kube-api-access-4gs2w" pod "58a200f0-5908-4059-999a-0704935ff375" (UID: "58a200f0-5908-4059-999a-0704935ff375"). InnerVolumeSpecName "kube-api-access-4gs2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:24:51.819578 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.819559 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-metrics-certs" pod "58a200f0-5908-4059-999a-0704935ff375" (UID: "58a200f0-5908-4059-999a-0704935ff375"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:51.819663 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.819592 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls" pod "58a200f0-5908-4059-999a-0704935ff375" (UID: "58a200f0-5908-4059-999a-0704935ff375"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:24:51.919016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.918903 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:51.919016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.918959 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gs2w\" (UniqueName: \"kubernetes.io/projected/58a200f0-5908-4059-999a-0704935ff375-kube-api-access-4gs2w\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:51.919016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:51.918970 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a200f0-5908-4059-999a-0704935ff375-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:24:52.145125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.145092 5139 generic.go:356] "Generic (PLEG): container finished" podID="58a200f0-5908-4059-999a-0704935ff375" containerID="10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492" exitCode=0 Oct 30 22:24:52.145283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.145162 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" Oct 30 22:24:52.145283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.145182 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" event={"ID":"58a200f0-5908-4059-999a-0704935ff375","Type":"ContainerDied","Data":"10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492"} Oct 30 22:24:52.145283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.145261 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j" event={"ID":"58a200f0-5908-4059-999a-0704935ff375","Type":"ContainerDied","Data":"90c43b2de1c09c8173557d489b9d3214dd3f3a758437470a84b0fdce439ecda5"} Oct 30 22:24:52.145436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.145285 5139 scope.go:122] "RemoveContainer" containerID="10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492" Oct 30 22:24:52.155258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.155242 5139 scope.go:122] "RemoveContainer" containerID="10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492" Oct 30 22:24:52.155539 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:24:52.155522 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492\": container with ID starting with 10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492 not found: ID does not exist" containerID="10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492" Oct 30 22:24:52.155601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.155546 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492"} err="failed to get container status \"10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492\": rpc error: code = NotFound desc = could not find container \"10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492\": container with ID starting with 10769b03796c6c8188eb7ca4b2de105c25e986217169fc7aec58a73c3dc1e492 not found: ID does not exist" Oct 30 22:24:52.300622 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.300594 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j"] Oct 30 22:24:52.304214 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:52.304186 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-ca6d0b-6b8947d774-mfp8j"] Oct 30 22:24:53.543578 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:53.543542 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a200f0-5908-4059-999a-0704935ff375" path="/var/lib/kubelet/pods/58a200f0-5908-4059-999a-0704935ff375/volumes" Oct 30 22:24:56.457614 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:56.457583 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5"] Oct 30 22:24:56.457972 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:24:56.457810 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" containerID="cri-o://39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b" gracePeriod=30 Oct 30 22:25:01.709146 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.709119 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 22:25:01.734178 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734108 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-istio-token\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-istio-token\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734178 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734156 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-kube-api-access-78xtc\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-kube-api-access-78xtc\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734378 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734196 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e263d7b9-e982-4966-a364-09b01ea59e50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263d7b9-e982-4966-a364-09b01ea59e50-istio-podinfo\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734378 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734235 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-socket\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734378 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734265 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-certs\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734378 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734364 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e263d7b9-e982-4966-a364-09b01ea59e50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263d7b9-e982-4966-a364-09b01ea59e50-istiod-ca-cert\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734503 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-credential-socket\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734548 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-data\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734563 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-socket" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:01.734619 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734615 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-envoy\") pod \"e263d7b9-e982-4966-a364-09b01ea59e50\" (UID: \"e263d7b9-e982-4966-a364-09b01ea59e50\") " Oct 30 22:25:01.734809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734628 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-certs" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:01.734809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734790 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-credential-socket" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:01.734905 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734799 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-data" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:01.734905 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.734810 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e263d7b9-e982-4966-a364-09b01ea59e50-istiod-ca-cert" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:01.735096 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.735079 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.735167 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.735102 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.735167 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.735115 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e263d7b9-e982-4966-a364-09b01ea59e50-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.735167 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.735129 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.735167 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.735142 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.736456 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.736429 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-kube-api-access-78xtc" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "kube-api-access-78xtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:01.736654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.736637 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-envoy" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:25:01.736898 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.736872 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e263d7b9-e982-4966-a364-09b01ea59e50-istio-podinfo" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:25:01.736994 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.736898 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-istio-token" pod "e263d7b9-e982-4966-a364-09b01ea59e50" (UID: "e263d7b9-e982-4966-a364-09b01ea59e50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:01.836767 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.836726 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e263d7b9-e982-4966-a364-09b01ea59e50-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.836767 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.836763 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.837008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.836787 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xtc\" (UniqueName: \"kubernetes.io/projected/e263d7b9-e982-4966-a364-09b01ea59e50-kube-api-access-78xtc\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:01.837008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:01.836802 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e263d7b9-e982-4966-a364-09b01ea59e50-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:02.206899 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.206860 5139 generic.go:356] "Generic (PLEG): container finished" podID="e263d7b9-e982-4966-a364-09b01ea59e50" containerID="39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b" exitCode=0 Oct 30 22:25:02.207050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.206925 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" event={"ID":"e263d7b9-e982-4966-a364-09b01ea59e50","Type":"ContainerDied","Data":"39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b"} Oct 30 22:25:02.207050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.206966 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" event={"ID":"e263d7b9-e982-4966-a364-09b01ea59e50","Type":"ContainerDied","Data":"a2490c0e9bac728d010edb0ba987a4775d9a4a280f04b13dd0431339c9e91e07"} Oct 30 22:25:02.207050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.206987 5139 scope.go:122] "RemoveContainer" containerID="39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b" Oct 30 22:25:02.207050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.206927 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5" Oct 30 22:25:02.218183 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.218165 5139 scope.go:122] "RemoveContainer" containerID="39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b" Oct 30 22:25:02.218487 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:25:02.218458 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b\": container with ID starting with 39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b not found: ID does not exist" containerID="39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b" Oct 30 22:25:02.218570 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.218496 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b"} err="failed to get container status \"39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b\": rpc error: code = NotFound desc = could not find container \"39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b\": container with ID starting with 39c2d9ad990a192068f13e28f71d494c1109637eba2e7ffdb6fecb28ad60639b not found: ID does not exist" Oct 30 22:25:02.361928 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.361894 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5"] Oct 30 22:25:02.408549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:02.408514 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-lpfj5"] Oct 30 22:25:03.544229 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:03.544195 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" path="/var/lib/kubelet/pods/e263d7b9-e982-4966-a364-09b01ea59e50/volumes" Oct 30 22:25:19.805402 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:25:19.805357 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:25:31.670172 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:31.670116 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc"] Oct 30 22:25:36.670572 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:36.670531 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" podUID="9cdd1485-7c01-476b-97b9-6155d59a1a76" containerName="payload-pre-processing" containerID="cri-o://974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3" gracePeriod=25 Oct 30 22:25:36.937555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:36.937495 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:25:37.060505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.060468 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdd1485-7c01-476b-97b9-6155d59a1a76-kube-api-access-hvmd5\" (UniqueName: \"kubernetes.io/projected/9cdd1485-7c01-476b-97b9-6155d59a1a76-kube-api-access-hvmd5\") pod \"9cdd1485-7c01-476b-97b9-6155d59a1a76\" (UID: \"9cdd1485-7c01-476b-97b9-6155d59a1a76\") " Oct 30 22:25:37.060694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.060578 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdd1485-7c01-476b-97b9-6155d59a1a76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd1485-7c01-476b-97b9-6155d59a1a76-plugins-config-volume\") pod \"9cdd1485-7c01-476b-97b9-6155d59a1a76\" (UID: \"9cdd1485-7c01-476b-97b9-6155d59a1a76\") " Oct 30 22:25:37.060971 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.060947 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdd1485-7c01-476b-97b9-6155d59a1a76-plugins-config-volume" pod "9cdd1485-7c01-476b-97b9-6155d59a1a76" (UID: "9cdd1485-7c01-476b-97b9-6155d59a1a76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:37.061095 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.061079 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd1485-7c01-476b-97b9-6155d59a1a76-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:37.062809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.062781 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdd1485-7c01-476b-97b9-6155d59a1a76-kube-api-access-hvmd5" pod "9cdd1485-7c01-476b-97b9-6155d59a1a76" (UID: "9cdd1485-7c01-476b-97b9-6155d59a1a76"). InnerVolumeSpecName "kube-api-access-hvmd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:37.162769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.162740 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvmd5\" (UniqueName: \"kubernetes.io/projected/9cdd1485-7c01-476b-97b9-6155d59a1a76-kube-api-access-hvmd5\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:37.400685 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.400647 5139 generic.go:356] "Generic (PLEG): container finished" podID="9cdd1485-7c01-476b-97b9-6155d59a1a76" containerID="974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3" exitCode=0 Oct 30 22:25:37.400917 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.400697 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" Oct 30 22:25:37.400917 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.400729 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" event={"ID":"9cdd1485-7c01-476b-97b9-6155d59a1a76","Type":"ContainerDied","Data":"974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3"} Oct 30 22:25:37.400917 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.400803 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc" event={"ID":"9cdd1485-7c01-476b-97b9-6155d59a1a76","Type":"ContainerDied","Data":"9d31fc6bd67e485480d7db7a71ae76604f14c799396f3e212942079705f8416d"} Oct 30 22:25:37.400917 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.400834 5139 scope.go:122] "RemoveContainer" containerID="974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3" Oct 30 22:25:37.410506 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.410488 5139 scope.go:122] "RemoveContainer" containerID="974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3" Oct 30 22:25:37.410779 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:25:37.410756 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3\": container with ID starting with 974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3 not found: ID does not exist" containerID="974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3" Oct 30 22:25:37.410839 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.410786 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3"} err="failed to get container status \"974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3\": rpc error: code = NotFound desc = could not find container \"974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3\": container with ID starting with 974b80057b996507aea5f2f3b415b15bc82da7e04f7d74678d481ad2064ae1e3 not found: ID does not exist" Oct 30 22:25:37.556272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.556226 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc"] Oct 30 22:25:37.602688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:37.602659 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-93131d-7c7b47dd47-9j7hc"] Oct 30 22:25:39.543795 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:39.543764 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdd1485-7c01-476b-97b9-6155d59a1a76" path="/var/lib/kubelet/pods/9cdd1485-7c01-476b-97b9-6155d59a1a76/volumes" Oct 30 22:25:49.804565 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:25:49.804529 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:25:53.798396 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:53.798356 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b"] Oct 30 22:25:53.798976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:53.798607 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" podUID="1423d7c3-8429-454a-a50e-b85602bb92d8" containerName="maas-api" containerID="cri-o://170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348" gracePeriod=30 Oct 30 22:25:54.066715 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.066692 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:25:54.146867 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.146835 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-metrics-certs\") pod \"1423d7c3-8429-454a-a50e-b85602bb92d8\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " Oct 30 22:25:54.147034 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.146937 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1423d7c3-8429-454a-a50e-b85602bb92d8-kube-api-access-6lzkr\" (UniqueName: \"kubernetes.io/projected/1423d7c3-8429-454a-a50e-b85602bb92d8-kube-api-access-6lzkr\") pod \"1423d7c3-8429-454a-a50e-b85602bb92d8\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " Oct 30 22:25:54.147034 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.147000 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-maas-api-tls\") pod \"1423d7c3-8429-454a-a50e-b85602bb92d8\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " Oct 30 22:25:54.147129 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.147114 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1423d7c3-8429-454a-a50e-b85602bb92d8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1423d7c3-8429-454a-a50e-b85602bb92d8-openshift-service-ca\") pod \"1423d7c3-8429-454a-a50e-b85602bb92d8\" (UID: \"1423d7c3-8429-454a-a50e-b85602bb92d8\") " Oct 30 22:25:54.147526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.147498 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1423d7c3-8429-454a-a50e-b85602bb92d8-openshift-service-ca" pod "1423d7c3-8429-454a-a50e-b85602bb92d8" (UID: "1423d7c3-8429-454a-a50e-b85602bb92d8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:25:54.147652 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.147608 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1423d7c3-8429-454a-a50e-b85602bb92d8-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.149048 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.149031 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-maas-api-tls" pod "1423d7c3-8429-454a-a50e-b85602bb92d8" (UID: "1423d7c3-8429-454a-a50e-b85602bb92d8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:54.149504 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.149483 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1423d7c3-8429-454a-a50e-b85602bb92d8-kube-api-access-6lzkr" pod "1423d7c3-8429-454a-a50e-b85602bb92d8" (UID: "1423d7c3-8429-454a-a50e-b85602bb92d8"). InnerVolumeSpecName "kube-api-access-6lzkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:25:54.149579 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.149511 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-metrics-certs" pod "1423d7c3-8429-454a-a50e-b85602bb92d8" (UID: "1423d7c3-8429-454a-a50e-b85602bb92d8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:25:54.249275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.249243 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.249275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.249270 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lzkr\" (UniqueName: \"kubernetes.io/projected/1423d7c3-8429-454a-a50e-b85602bb92d8-kube-api-access-6lzkr\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.249275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.249280 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1423d7c3-8429-454a-a50e-b85602bb92d8-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:25:54.497175 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.497141 5139 generic.go:356] "Generic (PLEG): container finished" podID="1423d7c3-8429-454a-a50e-b85602bb92d8" containerID="170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348" exitCode=0 Oct 30 22:25:54.497369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.497201 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" Oct 30 22:25:54.497369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.497219 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" event={"ID":"1423d7c3-8429-454a-a50e-b85602bb92d8","Type":"ContainerDied","Data":"170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348"} Oct 30 22:25:54.497369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.497289 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b" event={"ID":"1423d7c3-8429-454a-a50e-b85602bb92d8","Type":"ContainerDied","Data":"eda159042ac17ad48e89769ef08afe3cf21142a3576569a34f6f30e8e1aeff81"} Oct 30 22:25:54.497369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.497320 5139 scope.go:122] "RemoveContainer" containerID="170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348" Oct 30 22:25:54.516067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.508033 5139 scope.go:122] "RemoveContainer" containerID="170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348" Oct 30 22:25:54.516067 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:25:54.508322 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348\": container with ID starting with 170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348 not found: ID does not exist" containerID="170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348" Oct 30 22:25:54.516067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.508351 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348"} err="failed to get container status \"170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348\": rpc error: code = NotFound desc = could not find container \"170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348\": container with ID starting with 170b965c2db77acb4d9bd79841d56168c32278ca7228a748d58417e24b2b4348 not found: ID does not exist" Oct 30 22:25:54.652200 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.652168 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b"] Oct 30 22:25:54.657065 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:54.657039 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-fc3918-65745b5f87-v8f8b"] Oct 30 22:25:55.545843 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:25:55.545802 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1423d7c3-8429-454a-a50e-b85602bb92d8" path="/var/lib/kubelet/pods/1423d7c3-8429-454a-a50e-b85602bb92d8/volumes" Oct 30 22:26:02.976545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:02.976508 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx"] Oct 30 22:26:02.977017 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:02.976782 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" containerID="cri-o://a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed" gracePeriod=30 Oct 30 22:26:05.026750 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:05.026718 5139 patch_prober.go:29] interesting pod/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:26:05.027136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:05.026777 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:26:08.232498 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.232474 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:26:08.308562 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308538 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9d30451-0c2e-424d-a84e-2085794abd7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d30451-0c2e-424d-a84e-2085794abd7d-istiod-ca-cert\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.308706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308586 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-envoy\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.308706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308617 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-socket\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.308706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308672 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-kube-api-access-5hmnc\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-kube-api-access-5hmnc\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.308894 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308731 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-podinfo\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.308894 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308779 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-certs\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.308894 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308828 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-token\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-token\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.308894 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308867 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-credential-socket\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.309100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308939 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-data\") pod \"f9d30451-0c2e-424d-a84e-2085794abd7d\" (UID: \"f9d30451-0c2e-424d-a84e-2085794abd7d\") " Oct 30 22:26:08.309100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.308978 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-socket" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.309100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309025 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9d30451-0c2e-424d-a84e-2085794abd7d-istiod-ca-cert" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:26:08.309100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309061 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-certs" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.309306 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309213 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-credential-socket" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.309363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309325 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-data" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.309508 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309486 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.309638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309511 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.309638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309528 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.309638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309543 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.309638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.309558 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9d30451-0c2e-424d-a84e-2085794abd7d-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.311008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.310983 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-podinfo" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:26:08.311123 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.311102 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-envoy" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:26:08.311163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.311111 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-kube-api-access-5hmnc" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "kube-api-access-5hmnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:08.311163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.311149 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-token" pod "f9d30451-0c2e-424d-a84e-2085794abd7d" (UID: "f9d30451-0c2e-424d-a84e-2085794abd7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:26:08.410922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.410894 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.410922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.410917 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.410922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.410927 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9d30451-0c2e-424d-a84e-2085794abd7d-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.411199 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.410935 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hmnc\" (UniqueName: \"kubernetes.io/projected/f9d30451-0c2e-424d-a84e-2085794abd7d-kube-api-access-5hmnc\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:26:08.592927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.592853 5139 generic.go:356] "Generic (PLEG): container finished" podID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerID="a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed" exitCode=0 Oct 30 22:26:08.592927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.592906 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" Oct 30 22:26:08.593143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.592939 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" event={"ID":"f9d30451-0c2e-424d-a84e-2085794abd7d","Type":"ContainerDied","Data":"a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed"} Oct 30 22:26:08.593143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.592995 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx" event={"ID":"f9d30451-0c2e-424d-a84e-2085794abd7d","Type":"ContainerDied","Data":"a7e2f3bffac19e7018a342a8941d2d3854d0044824774611392a8adc931a3f89"} Oct 30 22:26:08.593143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.593018 5139 scope.go:122] "RemoveContainer" containerID="a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed" Oct 30 22:26:08.604359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.604341 5139 scope.go:122] "RemoveContainer" containerID="6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9" Oct 30 22:26:08.612727 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.612705 5139 scope.go:122] "RemoveContainer" containerID="a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed" Oct 30 22:26:08.613028 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:26:08.613009 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed\": container with ID starting with a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed not found: ID does not exist" containerID="a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed" Oct 30 22:26:08.613108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.613037 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed"} err="failed to get container status \"a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed\": rpc error: code = NotFound desc = could not find container \"a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed\": container with ID starting with a964687494233881df8e356562d807f4a9fe04468039e1170aa7bfac4dbb41ed not found: ID does not exist" Oct 30 22:26:08.613108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.613060 5139 scope.go:122] "RemoveContainer" containerID="6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9" Oct 30 22:26:08.613382 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:26:08.613343 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9\": container with ID starting with 6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9 not found: ID does not exist" containerID="6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9" Oct 30 22:26:08.613546 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.613380 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9"} err="failed to get container status \"6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9\": rpc error: code = NotFound desc = could not find container \"6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9\": container with ID starting with 6dbb2c319a5797d96f629a1ab4758f4f62a0fc8906f30fbb8dce2f611d98d4c9 not found: ID does not exist" Oct 30 22:26:08.748901 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.748875 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx"] Oct 30 22:26:08.756053 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:08.756018 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-fc3918-openshift-default-86455f997-9mnxx"] Oct 30 22:26:09.544250 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:09.544207 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" path="/var/lib/kubelet/pods/f9d30451-0c2e-424d-a84e-2085794abd7d/volumes" Oct 30 22:26:19.803952 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:26:19.803906 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:26:49.185892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.185859 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4"] Oct 30 22:26:49.186321 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186310 5139 cpu_manager.go:414] "containerMap: removing container" podUID="58a200f0-5908-4059-999a-0704935ff375" containerName="maas-api" Oct 30 22:26:49.186363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186321 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a200f0-5908-4059-999a-0704935ff375" containerName="maas-api" Oct 30 22:26:49.186363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186332 5139 cpu_manager.go:414] "containerMap: removing container" podUID="2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" containerName="maas-api" Oct 30 22:26:49.186363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186337 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" containerName="maas-api" Oct 30 22:26:49.186363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186355 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9cdd1485-7c01-476b-97b9-6155d59a1a76" containerName="payload-pre-processing" Oct 30 22:26:49.186363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186360 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdd1485-7c01-476b-97b9-6155d59a1a76" containerName="payload-pre-processing" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186369 5139 cpu_manager.go:414] "containerMap: removing container" podUID="1423d7c3-8429-454a-a50e-b85602bb92d8" containerName="maas-api" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186374 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1423d7c3-8429-454a-a50e-b85602bb92d8" containerName="maas-api" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186382 5139 cpu_manager.go:414] "containerMap: removing container" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186387 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186398 5139 cpu_manager.go:414] "containerMap: removing container" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186403 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186424 5139 cpu_manager.go:414] "containerMap: removing container" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186431 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186439 5139 cpu_manager.go:414] "containerMap: removing container" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186444 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186523 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a200f0-5908-4059-999a-0704935ff375" containerName="maas-api" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186531 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="2713cb58-da4f-430e-97f4-4c3cd1d5bf0d" containerName="maas-api" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186539 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdd1485-7c01-476b-97b9-6155d59a1a76" containerName="payload-pre-processing" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186546 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="1423d7c3-8429-454a-a50e-b85602bb92d8" containerName="maas-api" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186552 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="e263d7b9-e982-4966-a364-09b01ea59e50" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186559 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186565 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b8721f2-f926-4225-916b-a37cd5925fda" containerName="istio-proxy" Oct 30 22:26:49.186569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.186570 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9d30451-0c2e-424d-a84e-2085794abd7d" containerName="istio-proxy" Oct 30 22:26:49.191160 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.191140 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.244102 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.244075 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4d14e0-openshift-default-dockercfg-mqz9f\"" Oct 30 22:26:49.249727 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.249705 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.249829 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.249812 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.249871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.249844 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.249972 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.249956 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzrk8\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-kube-api-access-hzrk8\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.250017 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.250008 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.250052 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.250042 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.250143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.250128 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.250179 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.250165 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.250213 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.250187 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.293908 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.293878 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4"] Oct 30 22:26:49.350930 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.350903 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.350934 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.350969 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351048 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351082 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351327 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351140 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzrk8\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-kube-api-access-hzrk8\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351327 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351205 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351327 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351276 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351579 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351389 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351579 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351455 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351579 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351476 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351579 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351523 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.351828 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.351806 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.352144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.352116 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.353516 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.353493 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.353726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.353711 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.409965 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.409944 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.410144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.410127 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzrk8\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-kube-api-access-hzrk8\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.504333 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.504306 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:49.804762 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:26:49.804682 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:26:49.934996 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.934966 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4"] Oct 30 22:26:49.941495 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:26:49.941460 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fdbd86c_47fe_48ba_8e68_64706316e2c1.slice/crio-ec72fc042c98d46ea25aff7123646e8c80fb3ca31104f635af4021783b64d0aa WatchSource:0}: Error finding container ec72fc042c98d46ea25aff7123646e8c80fb3ca31104f635af4021783b64d0aa: Status 404 returned error can't find the container with id ec72fc042c98d46ea25aff7123646e8c80fb3ca31104f635af4021783b64d0aa Oct 30 22:26:49.943729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.943698 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:26:49.943798 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.943769 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:26:49.943836 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:49.943798 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:26:50.817453 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:50.817399 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" event={"ID":"8fdbd86c-47fe-48ba-8e68-64706316e2c1","Type":"ContainerStarted","Data":"99e6188aa46dc7834b6a1b1fe7f4426254e5c84a9b80eaf91a45b180dd6de7cc"} Oct 30 22:26:50.817847 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:50.817472 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" event={"ID":"8fdbd86c-47fe-48ba-8e68-64706316e2c1","Type":"ContainerStarted","Data":"ec72fc042c98d46ea25aff7123646e8c80fb3ca31104f635af4021783b64d0aa"} Oct 30 22:26:50.988973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:50.988903 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podStartSLOduration=1.9888821650000001 podStartE2EDuration="1.988882165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:50.987935935 +0000 UTC m=+2919.579221424" watchObservedRunningTime="2026-10-30 22:26:50.988882165 +0000 UTC m=+2919.580167655" Oct 30 22:26:51.505480 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:51.505449 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:51.506662 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:51.506635 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:26:51.506781 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:51.506680 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 22:26:52.440738 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:52.440695 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" podUID="3ab1a8fb7b394ed5c2ad3173d5294ab9" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:26:52.504934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:52.504898 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:26:52.505088 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:52.504959 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 22:26:53.505258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:53.505227 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:26:53.505774 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:53.505283 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 22:26:54.504680 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:54.504649 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:26:54.504868 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:54.504702 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 22:26:55.505956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:55.505915 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:26:55.506366 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:55.505976 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 22:26:56.504605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:56.504567 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" start-of-body= Oct 30 22:26:56.504837 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:56.504623 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.104:15021/healthz/ready\": dial tcp 10.134.0.104:15021: connect: connection refused" Oct 30 22:26:57.509284 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:57.509253 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:57.509782 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:57.509689 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:26:57.510441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:26:57.510397 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:27:09.618077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.618044 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8"] Oct 30 22:27:09.621788 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.621766 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.682813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.682785 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.682923 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.682820 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.682996 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.682976 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.683060 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.683043 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.683173 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.683150 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.683231 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.683192 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.683295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.683251 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.683435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.683389 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.683571 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.683465 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtxg4\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-kube-api-access-gtxg4\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.717383 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.717360 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3c4802-openshift-default-dockercfg-6gcqv\"" Oct 30 22:27:09.726942 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.726920 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8"] Oct 30 22:27:09.786985 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.786952 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787187 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.786999 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtxg4\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-kube-api-access-gtxg4\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787187 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.787055 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787187 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.787087 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.787227 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.787296 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.787330 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.787360 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.787391 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.787816 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.787784 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.788040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.788013 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.788226 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.788144 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.788226 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.788149 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.788470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.788449 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.790459 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.790399 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.790553 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.790467 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.844989 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.844958 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtxg4\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-kube-api-access-gtxg4\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.845892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.845864 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:09.935646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:09.935584 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:10.362079 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:10.362048 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8"] Oct 30 22:27:10.368712 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:27:10.368683 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod107566ff_9d1c_4ae2_be87_6adeaf7cbaa5.slice/crio-f43643c0d1ab393805532b84f352f41d8e49eb68f69ea77d620a43733cda74ab WatchSource:0}: Error finding container f43643c0d1ab393805532b84f352f41d8e49eb68f69ea77d620a43733cda74ab: Status 404 returned error can't find the container with id f43643c0d1ab393805532b84f352f41d8e49eb68f69ea77d620a43733cda74ab Oct 30 22:27:10.371309 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:10.371277 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:10.371375 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:10.371346 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:10.371431 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:10.371376 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:27:10.925631 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:10.925592 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" event={"ID":"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5","Type":"ContainerStarted","Data":"acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb"} Oct 30 22:27:10.926012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:10.925656 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" event={"ID":"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5","Type":"ContainerStarted","Data":"f43643c0d1ab393805532b84f352f41d8e49eb68f69ea77d620a43733cda74ab"} Oct 30 22:27:10.935880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:10.935858 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:11.042342 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:11.042291 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podStartSLOduration=2.042276839 podStartE2EDuration="2.042276839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:11.04100129 +0000 UTC m=+2939.632286780" watchObservedRunningTime="2026-10-30 22:27:11.042276839 +0000 UTC m=+2939.633562325" Oct 30 22:27:11.937854 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:11.937820 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:27:11.938228 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:11.937880 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:27:12.936494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:12.936459 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:27:12.936675 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:12.936513 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:27:13.936756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:13.936718 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:27:13.937220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:13.936776 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:27:14.936242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:14.936205 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:27:14.936437 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:14.936261 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:27:15.936312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:15.936274 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:27:15.936759 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:15.936336 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:27:16.936272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:16.936236 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" start-of-body= Oct 30 22:27:16.936473 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:16.936285 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.105:15021/healthz/ready\": dial tcp 10.134.0.105:15021: connect: connection refused" Oct 30 22:27:17.940949 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:17.940920 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:17.941431 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:17.941373 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:17.942377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:17.942352 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:27:19.804145 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:27:19.804107 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:27:49.805211 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:27:49.805173 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:27:56.772978 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:27:56.772551 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd"] Oct 30 22:28:01.773332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:01.773287 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" podUID="dc491e22-ff69-4d08-a8da-40c265bb2d02" containerName="payload-pre-processing" containerID="cri-o://fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a" gracePeriod=25 Oct 30 22:28:02.299574 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.299540 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h"] Oct 30 22:28:02.299776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.299754 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" podUID="5d3a9723-dbd9-46bf-be14-2bf7c390030f" containerName="maas-api" containerID="cri-o://cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b" gracePeriod=30 Oct 30 22:28:02.575870 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.575788 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:28:02.619689 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.619656 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls\") pod \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " Oct 30 22:28:02.619863 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.619763 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-metrics-certs\") pod \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " Oct 30 22:28:02.619907 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.619863 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3a9723-dbd9-46bf-be14-2bf7c390030f-kube-api-access-kwd2z\" (UniqueName: \"kubernetes.io/projected/5d3a9723-dbd9-46bf-be14-2bf7c390030f-kube-api-access-kwd2z\") pod \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " Oct 30 22:28:02.619907 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.619891 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3a9723-dbd9-46bf-be14-2bf7c390030f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a9723-dbd9-46bf-be14-2bf7c390030f-openshift-service-ca\") pod \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\" (UID: \"5d3a9723-dbd9-46bf-be14-2bf7c390030f\") " Oct 30 22:28:02.620314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.620285 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3a9723-dbd9-46bf-be14-2bf7c390030f-openshift-service-ca" pod "5d3a9723-dbd9-46bf-be14-2bf7c390030f" (UID: "5d3a9723-dbd9-46bf-be14-2bf7c390030f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:02.620662 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.620636 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d3a9723-dbd9-46bf-be14-2bf7c390030f-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:02.622011 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.621983 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-metrics-certs" pod "5d3a9723-dbd9-46bf-be14-2bf7c390030f" (UID: "5d3a9723-dbd9-46bf-be14-2bf7c390030f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:02.622248 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.622223 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls" pod "5d3a9723-dbd9-46bf-be14-2bf7c390030f" (UID: "5d3a9723-dbd9-46bf-be14-2bf7c390030f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:28:02.622248 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.622227 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3a9723-dbd9-46bf-be14-2bf7c390030f-kube-api-access-kwd2z" pod "5d3a9723-dbd9-46bf-be14-2bf7c390030f" (UID: "5d3a9723-dbd9-46bf-be14-2bf7c390030f"). InnerVolumeSpecName "kube-api-access-kwd2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:02.721880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.721834 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:02.721880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.721879 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwd2z\" (UniqueName: \"kubernetes.io/projected/5d3a9723-dbd9-46bf-be14-2bf7c390030f-kube-api-access-kwd2z\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:02.722089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:02.721896 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5d3a9723-dbd9-46bf-be14-2bf7c390030f-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:03.211527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.211491 5139 generic.go:356] "Generic (PLEG): container finished" podID="5d3a9723-dbd9-46bf-be14-2bf7c390030f" containerID="cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b" exitCode=0 Oct 30 22:28:03.211982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.211542 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" Oct 30 22:28:03.211982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.211560 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" event={"ID":"5d3a9723-dbd9-46bf-be14-2bf7c390030f","Type":"ContainerDied","Data":"cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b"} Oct 30 22:28:03.211982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.211630 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h" event={"ID":"5d3a9723-dbd9-46bf-be14-2bf7c390030f","Type":"ContainerDied","Data":"d852660e46381a91c07de51622feca29a964fe603bd9aa32183fe89fe28b9a92"} Oct 30 22:28:03.211982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.211654 5139 scope.go:122] "RemoveContainer" containerID="cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b" Oct 30 22:28:03.222246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.222225 5139 scope.go:122] "RemoveContainer" containerID="cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b" Oct 30 22:28:03.222572 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:28:03.222553 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b\": container with ID starting with cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b not found: ID does not exist" containerID="cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b" Oct 30 22:28:03.222641 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.222583 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b"} err="failed to get container status \"cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b\": rpc error: code = NotFound desc = could not find container \"cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b\": container with ID starting with cba5d17c2bd64b6396cea3cd0a936fd3637ae4856ab6c19099e1c3ac295eac5b not found: ID does not exist" Oct 30 22:28:03.370984 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.370951 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h"] Oct 30 22:28:03.385158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.385098 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d2f1cf-66fc75cd49-7495h"] Oct 30 22:28:03.546284 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:03.546246 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3a9723-dbd9-46bf-be14-2bf7c390030f" path="/var/lib/kubelet/pods/5d3a9723-dbd9-46bf-be14-2bf7c390030f/volumes" Oct 30 22:28:10.419320 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:10.419284 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" podUID="dc491e22-ff69-4d08-a8da-40c265bb2d02" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:11.962599 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:11.962562 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr"] Oct 30 22:28:11.963031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:11.962782 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" containerID="cri-o://ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b" gracePeriod=30 Oct 30 22:28:12.643188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:12.643113 5139 patch_prober.go:29] interesting pod/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:28:12.643188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:12.643173 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:28:17.227580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.227513 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:28:17.287402 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.287363 5139 generic.go:356] "Generic (PLEG): container finished" podID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerID="ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b" exitCode=0 Oct 30 22:28:17.287625 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.287451 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" Oct 30 22:28:17.287625 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.287468 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" event={"ID":"5eda784e-ba36-4d39-9c85-8e0d63656568","Type":"ContainerDied","Data":"ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b"} Oct 30 22:28:17.287625 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.287539 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr" event={"ID":"5eda784e-ba36-4d39-9c85-8e0d63656568","Type":"ContainerDied","Data":"92d3e56d9a2f85c3e7c758a9bbc0bc62f3147ffe4cacb4abc085d60e51077be1"} Oct 30 22:28:17.287625 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.287573 5139 scope.go:122] "RemoveContainer" containerID="ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b" Oct 30 22:28:17.296551 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.296528 5139 scope.go:122] "RemoveContainer" containerID="5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed" Oct 30 22:28:17.305011 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.304990 5139 scope.go:122] "RemoveContainer" containerID="ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b" Oct 30 22:28:17.305308 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:28:17.305288 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b\": container with ID starting with ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b not found: ID does not exist" containerID="ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b" Oct 30 22:28:17.305363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.305317 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b"} err="failed to get container status \"ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b\": rpc error: code = NotFound desc = could not find container \"ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b\": container with ID starting with ce6365017d65598b6da99f8618c50a3ca240919e9667b5ce0d7f7fff297e7f7b not found: ID does not exist" Oct 30 22:28:17.305363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.305338 5139 scope.go:122] "RemoveContainer" containerID="5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed" Oct 30 22:28:17.305625 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:28:17.305609 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed\": container with ID starting with 5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed not found: ID does not exist" containerID="5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed" Oct 30 22:28:17.305673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.305632 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed"} err="failed to get container status \"5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed\": rpc error: code = NotFound desc = could not find container \"5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed\": container with ID starting with 5deba27d0d64ff10856baba1b507b8913e972163b60f8ec3d3615d8e5708e9ed not found: ID does not exist" Oct 30 22:28:17.383764 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.383730 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-certs\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.383940 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.383827 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-kube-api-access-vlfxg\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-kube-api-access-vlfxg\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.383940 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.383857 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-data\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.383940 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.383927 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eda784e-ba36-4d39-9c85-8e0d63656568-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda784e-ba36-4d39-9c85-8e0d63656568-istiod-ca-cert\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.384101 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.383973 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-token\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-token\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.384101 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384019 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-credential-socket\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.384101 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384063 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-certs" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:17.384101 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384074 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-socket\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.384318 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384109 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-podinfo\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.384318 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384140 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-envoy\") pod \"5eda784e-ba36-4d39-9c85-8e0d63656568\" (UID: \"5eda784e-ba36-4d39-9c85-8e0d63656568\") " Oct 30 22:28:17.384318 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384240 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-data" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:17.384318 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384277 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-credential-socket" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:17.384547 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384343 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eda784e-ba36-4d39-9c85-8e0d63656568-istiod-ca-cert" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:17.384547 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384477 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-socket" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:17.384810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384792 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.384890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384816 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.384890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384831 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.384890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384843 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5eda784e-ba36-4d39-9c85-8e0d63656568-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.384890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.384857 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.386312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.386289 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-podinfo" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:28:17.386568 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.386544 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-envoy" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:28:17.386749 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.386728 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-kube-api-access-vlfxg" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "kube-api-access-vlfxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:17.386811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.386760 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-token" pod "5eda784e-ba36-4d39-9c85-8e0d63656568" (UID: "5eda784e-ba36-4d39-9c85-8e0d63656568"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:17.486603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.486523 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.486603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.486553 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.486603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.486564 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlfxg\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-kube-api-access-vlfxg\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.486603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.486574 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5eda784e-ba36-4d39-9c85-8e0d63656568-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:17.695925 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.695891 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr"] Oct 30 22:28:17.698347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:17.698319 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d2f1cf-openshift-default-7bddd68c6f-vj7xr"] Oct 30 22:28:19.542317 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:19.542265 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" path="/var/lib/kubelet/pods/5eda784e-ba36-4d39-9c85-8e0d63656568/volumes" Oct 30 22:28:19.804491 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:28:19.804389 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:28:20.419384 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:20.419341 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" podUID="dc491e22-ff69-4d08-a8da-40c265bb2d02" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:28:22.036080 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.036015 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:28:22.036538 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.036292 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc491e22-ff69-4d08-a8da-40c265bb2d02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc491e22-ff69-4d08-a8da-40c265bb2d02-plugins-config-volume\") pod \"dc491e22-ff69-4d08-a8da-40c265bb2d02\" (UID: \"dc491e22-ff69-4d08-a8da-40c265bb2d02\") " Oct 30 22:28:22.036538 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.036333 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc491e22-ff69-4d08-a8da-40c265bb2d02-kube-api-access-z75xz\" (UniqueName: \"kubernetes.io/projected/dc491e22-ff69-4d08-a8da-40c265bb2d02-kube-api-access-z75xz\") pod \"dc491e22-ff69-4d08-a8da-40c265bb2d02\" (UID: \"dc491e22-ff69-4d08-a8da-40c265bb2d02\") " Oct 30 22:28:22.036890 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.036868 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc491e22-ff69-4d08-a8da-40c265bb2d02-plugins-config-volume" pod "dc491e22-ff69-4d08-a8da-40c265bb2d02" (UID: "dc491e22-ff69-4d08-a8da-40c265bb2d02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:28:22.038434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.038392 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc491e22-ff69-4d08-a8da-40c265bb2d02-kube-api-access-z75xz" pod "dc491e22-ff69-4d08-a8da-40c265bb2d02" (UID: "dc491e22-ff69-4d08-a8da-40c265bb2d02"). InnerVolumeSpecName "kube-api-access-z75xz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:28:22.137892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.137849 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc491e22-ff69-4d08-a8da-40c265bb2d02-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:22.137892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.137896 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z75xz\" (UniqueName: \"kubernetes.io/projected/dc491e22-ff69-4d08-a8da-40c265bb2d02-kube-api-access-z75xz\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:28:22.329950 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.329869 5139 generic.go:356] "Generic (PLEG): container finished" podID="dc491e22-ff69-4d08-a8da-40c265bb2d02" containerID="fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a" exitCode=0 Oct 30 22:28:22.330091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.329966 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" event={"ID":"dc491e22-ff69-4d08-a8da-40c265bb2d02","Type":"ContainerDied","Data":"fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a"} Oct 30 22:28:22.330091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.330042 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" event={"ID":"dc491e22-ff69-4d08-a8da-40c265bb2d02","Type":"ContainerDied","Data":"0b3da50b09bebc287ba1afeba00689a6f6873043b8f119b2f6aacc625743c109"} Oct 30 22:28:22.330091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.330069 5139 scope.go:122] "RemoveContainer" containerID="fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a" Oct 30 22:28:22.330236 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.330223 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd" Oct 30 22:28:22.339822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.339799 5139 scope.go:122] "RemoveContainer" containerID="fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a" Oct 30 22:28:22.340087 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:28:22.340065 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a\": container with ID starting with fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a not found: ID does not exist" containerID="fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a" Oct 30 22:28:22.340157 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.340099 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a"} err="failed to get container status \"fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a\": rpc error: code = NotFound desc = could not find container \"fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a\": container with ID starting with fdad415ab13690d178ceff8ca836e3aec5bbfd10d7d375307b511d6c4ca5cf7a not found: ID does not exist" Oct 30 22:28:22.487172 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.487138 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd"] Oct 30 22:28:22.495085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:22.495054 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f7c488-664c5d8875-bjnxd"] Oct 30 22:28:23.544108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:23.544071 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc491e22-ff69-4d08-a8da-40c265bb2d02" path="/var/lib/kubelet/pods/dc491e22-ff69-4d08-a8da-40c265bb2d02/volumes" Oct 30 22:28:49.804490 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:28:49.804455 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:28:50.136662 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.136592 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll"] Oct 30 22:28:50.137044 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137030 5139 cpu_manager.go:414] "containerMap: removing container" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" Oct 30 22:28:50.137093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137044 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" Oct 30 22:28:50.137093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137062 5139 cpu_manager.go:414] "containerMap: removing container" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" Oct 30 22:28:50.137093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137067 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" Oct 30 22:28:50.137093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137080 5139 cpu_manager.go:414] "containerMap: removing container" podUID="dc491e22-ff69-4d08-a8da-40c265bb2d02" containerName="payload-pre-processing" Oct 30 22:28:50.137093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137085 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc491e22-ff69-4d08-a8da-40c265bb2d02" containerName="payload-pre-processing" Oct 30 22:28:50.137093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137093 5139 cpu_manager.go:414] "containerMap: removing container" podUID="5d3a9723-dbd9-46bf-be14-2bf7c390030f" containerName="maas-api" Oct 30 22:28:50.137282 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137098 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3a9723-dbd9-46bf-be14-2bf7c390030f" containerName="maas-api" Oct 30 22:28:50.137282 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137167 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" Oct 30 22:28:50.137282 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137176 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eda784e-ba36-4d39-9c85-8e0d63656568" containerName="istio-proxy" Oct 30 22:28:50.137282 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137184 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc491e22-ff69-4d08-a8da-40c265bb2d02" containerName="payload-pre-processing" Oct 30 22:28:50.137282 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.137189 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3a9723-dbd9-46bf-be14-2bf7c390030f" containerName="maas-api" Oct 30 22:28:50.140274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.140256 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.183026 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.183001 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg"] Oct 30 22:28:50.186635 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.186620 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.240561 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.240538 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll"] Oct 30 22:28:50.248092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248053 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.248220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248127 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.248303 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248238 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4md2x\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-kube-api-access-4md2x\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.248303 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248277 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.248438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248337 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.248438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248383 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.248438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248409 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.248592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248459 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.248592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.248498 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.296856 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.296828 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg"] Oct 30 22:28:50.349530 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.349486 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.349686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.349541 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada185ee-3ddd-43ac-bdfa-b29e211193be-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.349686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.349570 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.349686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.349594 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.349686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.349618 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.349839 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.349683 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.349839 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.349733 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.349839 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.349809 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350046 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.350139 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350099 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350198 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350149 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d88t6\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-kube-api-access-d88t6\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.350260 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350201 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.350260 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350225 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350260 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350230 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.350444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350280 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350309 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350357 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350373 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350400 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.350698 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350492 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4md2x\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-kube-api-access-4md2x\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350698 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350538 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.350698 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350570 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.350848 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.350829 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.352108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.352092 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.352746 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.352728 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.408436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.408361 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.408806 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.408784 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4md2x\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-kube-api-access-4md2x\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.451505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451473 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.451505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451506 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.451726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451544 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.451726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451573 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.451726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451623 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada185ee-3ddd-43ac-bdfa-b29e211193be-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.451726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451651 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.451726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451681 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.452008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451803 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.452008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451847 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d88t6\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-kube-api-access-d88t6\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.452008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451879 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.452008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.451923 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.452235 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.452215 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.452319 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.452298 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.452370 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.452301 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada185ee-3ddd-43ac-bdfa-b29e211193be-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.452469 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.452449 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:50.454155 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.454126 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.454458 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.454411 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.509936 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.509901 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.510686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.510662 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d88t6\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-kube-api-access-d88t6\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.798560 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.798532 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:50.887791 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.887757 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll"] Oct 30 22:28:50.893273 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:28:50.893247 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f1e1e49_35b4_4af9_ac05_3ac2c2ac243f.slice/crio-b4d5a4760f5d16d3c287e281b421a0f3943f3e9efc05aaf5edb03cdf2bb6ceaa WatchSource:0}: Error finding container b4d5a4760f5d16d3c287e281b421a0f3943f3e9efc05aaf5edb03cdf2bb6ceaa: Status 404 returned error can't find the container with id b4d5a4760f5d16d3c287e281b421a0f3943f3e9efc05aaf5edb03cdf2bb6ceaa Oct 30 22:28:50.895986 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.895948 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:50.896087 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.896029 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:50.896155 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:50.896101 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:51.176051 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.175956 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg"] Oct 30 22:28:51.181362 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:28:51.181332 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada185ee_3ddd_43ac_bdfa_b29e211193be.slice/crio-954340746a1cc42d86ec1ae70eb98d5fb96abff87c30b7f5d756b0fb75214824 WatchSource:0}: Error finding container 954340746a1cc42d86ec1ae70eb98d5fb96abff87c30b7f5d756b0fb75214824: Status 404 returned error can't find the container with id 954340746a1cc42d86ec1ae70eb98d5fb96abff87c30b7f5d756b0fb75214824 Oct 30 22:28:51.184300 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.184258 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:51.184399 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.184349 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:51.184458 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.184393 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:51.498430 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.498386 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" event={"ID":"ada185ee-3ddd-43ac-bdfa-b29e211193be","Type":"ContainerStarted","Data":"068a5630d1203cbb62548ed309c33a6b9299a39299c4e6c6c1192798806ad214"} Oct 30 22:28:51.498635 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.498460 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" event={"ID":"ada185ee-3ddd-43ac-bdfa-b29e211193be","Type":"ContainerStarted","Data":"954340746a1cc42d86ec1ae70eb98d5fb96abff87c30b7f5d756b0fb75214824"} Oct 30 22:28:51.500694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.500666 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" event={"ID":"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f","Type":"ContainerStarted","Data":"623117ebce44bddf4661ac3cd87bb284ccf31a0b2e804c1e077d24afee4b0e9e"} Oct 30 22:28:51.500835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.500719 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" event={"ID":"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f","Type":"ContainerStarted","Data":"b4d5a4760f5d16d3c287e281b421a0f3943f3e9efc05aaf5edb03cdf2bb6ceaa"} Oct 30 22:28:51.614432 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.614360 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" podStartSLOduration=1.614346364 podStartE2EDuration="1.614346364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:51.612811045 +0000 UTC m=+3040.204096538" watchObservedRunningTime="2026-10-30 22:28:51.614346364 +0000 UTC m=+3040.205631850" Oct 30 22:28:51.810395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:51.810305 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:52.441544 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:52.441502 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" podUID="3ab1a8fb7b394ed5c2ad3173d5294ab9" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 22:28:52.453088 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:52.453061 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:52.455708 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:52.455688 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:28:52.455841 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:52.455819 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:28:52.818521 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:52.815172 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:28:52.818521 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:52.815227 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 22:28:53.453286 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:53.453248 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:28:53.453722 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:53.453301 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:28:53.802198 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:53.801756 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:28:53.802198 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:53.801833 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 22:28:54.453016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.452981 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:28:54.453310 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.453280 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:28:54.510310 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.510240 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" podStartSLOduration=4.510218909 podStartE2EDuration="4.510218909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:51.809568584 +0000 UTC m=+3040.400854071" watchObservedRunningTime="2026-10-30 22:28:54.510218909 +0000 UTC m=+3043.101504397" Oct 30 22:28:54.536553 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.536491 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng"] Oct 30 22:28:54.543140 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.543102 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:54.548071 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.547203 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng\" (UID: \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:54.548071 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.547384 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlxkn\" (UniqueName: \"kubernetes.io/projected/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-kube-api-access-rlxkn\") pod \"payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng\" (UID: \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:54.584439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.580027 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng"] Oct 30 22:28:54.626019 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.625987 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5e884c-dockercfg-zkw6b\"" Oct 30 22:28:54.627057 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.627031 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5e884c\"" Oct 30 22:28:54.656434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.650925 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlxkn\" (UniqueName: \"kubernetes.io/projected/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-kube-api-access-rlxkn\") pod \"payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng\" (UID: \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:54.656434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.651009 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng\" (UID: \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:54.656434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.651666 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng\" (UID: \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:54.716489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.715980 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlxkn\" (UniqueName: \"kubernetes.io/projected/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-kube-api-access-rlxkn\") pod \"payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng\" (UID: \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:54.807441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.807209 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:28:54.807441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.807279 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 22:28:54.883441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:54.881708 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:55.301752 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:28:55.301680 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a0e33b4_d4e4_468f_b9b4_c78654a41f13.slice/crio-a44e4772f2f88225a7546e4582be620bef033c9b3e91b7b26c99f83ae6022153 WatchSource:0}: Error finding container a44e4772f2f88225a7546e4582be620bef033c9b3e91b7b26c99f83ae6022153: Status 404 returned error can't find the container with id a44e4772f2f88225a7546e4582be620bef033c9b3e91b7b26c99f83ae6022153 Oct 30 22:28:55.305447 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.303449 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng"] Oct 30 22:28:55.434364 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.434338 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8"] Oct 30 22:28:55.438168 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.438149 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.452810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.452784 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:28:55.452926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.452832 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:28:55.458180 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458160 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.458537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458195 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.458537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458228 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snv5f\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-kube-api-access-snv5f\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.458537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458258 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.458537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458302 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.458537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458334 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.458537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458358 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca472a-aeec-4144-a7bf-516da8567642-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.458537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458373 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca472a-aeec-4144-a7bf-516da8567642-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.458537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.458410 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.519042 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.517686 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8"] Oct 30 22:28:55.557203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.557113 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" event={"ID":"4a0e33b4-d4e4-468f-b9b4-c78654a41f13","Type":"ContainerStarted","Data":"6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a"} Oct 30 22:28:55.557485 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.557448 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" event={"ID":"4a0e33b4-d4e4-468f-b9b4-c78654a41f13","Type":"ContainerStarted","Data":"a44e4772f2f88225a7546e4582be620bef033c9b3e91b7b26c99f83ae6022153"} Oct 30 22:28:55.557638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.557625 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:28:55.559997 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.559973 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560015 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560039 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca472a-aeec-4144-a7bf-516da8567642-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560054 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca472a-aeec-4144-a7bf-516da8567642-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560083 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560146 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560168 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560199 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snv5f\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-kube-api-access-snv5f\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560231 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.560695 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.560599 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.561395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.561368 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.561618 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.561598 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.561855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.561837 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.570445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.562409 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca472a-aeec-4144-a7bf-516da8567642-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.595459 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.570775 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.595459 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.575929 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca472a-aeec-4144-a7bf-516da8567642-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.625657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.625614 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.626050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.626028 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snv5f\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-kube-api-access-snv5f\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.750852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.750821 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:55.804445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.799747 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:28:55.804445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:55.799800 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 22:28:56.226393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.226290 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" podStartSLOduration=2.226276892 podStartE2EDuration="2.226276892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:55.705741138 +0000 UTC m=+3044.297026628" watchObservedRunningTime="2026-10-30 22:28:56.226276892 +0000 UTC m=+3044.817562378" Oct 30 22:28:56.227004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.226984 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8"] Oct 30 22:28:56.232925 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:28:56.232897 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0ca472a_aeec_4144_a7bf_516da8567642.slice/crio-d05fe9c29e2081d6472d0dba089b96366dcf8c025b94c3d34335051dfec6bdd4 WatchSource:0}: Error finding container d05fe9c29e2081d6472d0dba089b96366dcf8c025b94c3d34335051dfec6bdd4: Status 404 returned error can't find the container with id d05fe9c29e2081d6472d0dba089b96366dcf8c025b94c3d34335051dfec6bdd4 Oct 30 22:28:56.235457 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.235405 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:56.235564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.235513 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:56.235630 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.235560 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:56.311059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.310991 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz"] Oct 30 22:28:56.317464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.317046 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.374518 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.372333 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9a5623-openshift-default-dockercfg-gltcf\"" Oct 30 22:28:56.391445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.387797 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz"] Oct 30 22:28:56.425712 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.425678 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.426012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.425996 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.426139 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.426111 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.426233 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.426222 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495e4ea7-3763-412a-a131-40f30023e5da-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.426394 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.426381 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.426568 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.426553 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.426683 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.426673 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.426807 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.426796 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46bb5\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-kube-api-access-46bb5\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.426955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.426944 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495e4ea7-3763-412a-a131-40f30023e5da-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.453009 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.452966 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:28:56.453289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.453254 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:28:56.527801 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.527769 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.528273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.527828 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.528273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.527848 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.528273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.527877 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495e4ea7-3763-412a-a131-40f30023e5da-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.528273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.527925 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.528273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.527969 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.528273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.527984 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.528273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.528040 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46bb5\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-kube-api-access-46bb5\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.528273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.528084 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495e4ea7-3763-412a-a131-40f30023e5da-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.533592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.529153 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.533592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.529263 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495e4ea7-3763-412a-a131-40f30023e5da-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.533592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.529538 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.533592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.530097 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.533592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.530618 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.533592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.531749 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495e4ea7-3763-412a-a131-40f30023e5da-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.534433 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.534391 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.581440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.570119 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" event={"ID":"a0ca472a-aeec-4144-a7bf-516da8567642","Type":"ContainerStarted","Data":"d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d"} Oct 30 22:28:56.581440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.570177 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" event={"ID":"a0ca472a-aeec-4144-a7bf-516da8567642","Type":"ContainerStarted","Data":"d05fe9c29e2081d6472d0dba089b96366dcf8c025b94c3d34335051dfec6bdd4"} Oct 30 22:28:56.611659 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.611625 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46bb5\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-kube-api-access-46bb5\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.615045 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.615017 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.650259 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.650225 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:56.705398 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.704506 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podStartSLOduration=1.704483366 podStartE2EDuration="1.704483366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:56.703812112 +0000 UTC m=+3045.295097601" watchObservedRunningTime="2026-10-30 22:28:56.704483366 +0000 UTC m=+3045.295768873" Oct 30 22:28:56.750913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.750880 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:28:56.805463 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.800267 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:28:56.805463 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:56.800325 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 22:28:57.192053 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.191956 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz"] Oct 30 22:28:57.197322 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:28:57.197295 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod495e4ea7_3763_412a_a131_40f30023e5da.slice/crio-636f7dd098726d0e535fe2d66fcb4aa20a9b81c0463b66c2c457a8de141d1297 WatchSource:0}: Error finding container 636f7dd098726d0e535fe2d66fcb4aa20a9b81c0463b66c2c457a8de141d1297: Status 404 returned error can't find the container with id 636f7dd098726d0e535fe2d66fcb4aa20a9b81c0463b66c2c457a8de141d1297 Oct 30 22:28:57.200561 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.200526 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:57.200652 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.200596 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:57.200652 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.200632 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:28:57.453000 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.452901 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 30 22:28:57.453000 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.452970 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 30 22:28:57.602034 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.601974 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" event={"ID":"495e4ea7-3763-412a-a131-40f30023e5da","Type":"ContainerStarted","Data":"38d77984df9779db5c423b49edbfbf16183526a47f5d365342bfe6e6e59143bd"} Oct 30 22:28:57.602557 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.602061 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" event={"ID":"495e4ea7-3763-412a-a131-40f30023e5da","Type":"ContainerStarted","Data":"636f7dd098726d0e535fe2d66fcb4aa20a9b81c0463b66c2c457a8de141d1297"} Oct 30 22:28:57.653040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.651573 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:28:57.719089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.718965 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podStartSLOduration=1.718947441 podStartE2EDuration="1.718947441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:57.717025689 +0000 UTC m=+3046.308311176" watchObservedRunningTime="2026-10-30 22:28:57.718947441 +0000 UTC m=+3046.310232931" Oct 30 22:28:57.752952 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.752921 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:28:57.753129 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.752980 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:28:57.800879 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.800833 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 30 22:28:57.801976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:57.801937 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 30 22:28:58.653252 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:58.652898 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:28:58.653252 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:58.652962 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:28:58.759047 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:58.753724 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:28:58.759047 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:58.753816 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:28:59.009439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:58.999784 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:59.009439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.001401 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:59.009728 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.009634 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:28:59.137348 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.136799 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:59.137538 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.137466 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:59.138434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.138387 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:28:59.651005 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.650973 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:28:59.651197 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.651020 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:28:59.751174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.751133 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:28:59.751592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:28:59.751193 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:29:00.650688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:00.650654 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:29:00.650866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:00.650703 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:29:00.751262 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:00.751225 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:29:00.751643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:00.751288 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:29:01.483442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.481753 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj"] Oct 30 22:29:01.483442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.481969 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" podUID="3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" containerName="maas-api" containerID="cri-o://43b5496845c699312ab20d6fb6aaa36ec6074006cac36d96c098906c650f121c" gracePeriod=30 Oct 30 22:29:01.648071 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.647943 5139 generic.go:356] "Generic (PLEG): container finished" podID="3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" containerID="43b5496845c699312ab20d6fb6aaa36ec6074006cac36d96c098906c650f121c" exitCode=0 Oct 30 22:29:01.648071 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.648033 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" event={"ID":"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f","Type":"ContainerDied","Data":"43b5496845c699312ab20d6fb6aaa36ec6074006cac36d96c098906c650f121c"} Oct 30 22:29:01.651223 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.651197 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:29:01.651338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.651265 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:29:01.752913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.751154 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:29:01.752913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.751210 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:29:01.957555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:01.957531 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:29:02.058911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.058823 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls\") pod \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " Oct 30 22:29:02.058911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.058886 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-kube-api-access-vcnxb\" (UniqueName: \"kubernetes.io/projected/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-kube-api-access-vcnxb\") pod \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " Oct 30 22:29:02.059165 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.058985 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-metrics-certs\") pod \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " Oct 30 22:29:02.059165 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.059016 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-openshift-service-ca\") pod \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\" (UID: \"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f\") " Oct 30 22:29:02.059459 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.059405 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-openshift-service-ca" pod "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" (UID: "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:02.060986 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.060955 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls" pod "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" (UID: "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:02.061200 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.061176 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-metrics-certs" pod "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" (UID: "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:29:02.061249 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.061227 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-kube-api-access-vcnxb" pod "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" (UID: "3184ce2a-e642-4ac6-9bc8-fe1d65caec2f"). InnerVolumeSpecName "kube-api-access-vcnxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:02.160650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.160617 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:02.160650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.160649 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcnxb\" (UniqueName: \"kubernetes.io/projected/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-kube-api-access-vcnxb\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:02.160862 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.160663 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:02.160862 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.160677 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:02.655563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.651936 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:29:02.655563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.652010 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:29:02.675442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.673909 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" event={"ID":"3184ce2a-e642-4ac6-9bc8-fe1d65caec2f","Type":"ContainerDied","Data":"99bb1fe079b49d1bf19b8a8ed12719f97d4650e9ca378a29d7ac150ec96e9851"} Oct 30 22:29:02.675442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.673992 5139 scope.go:122] "RemoveContainer" containerID="43b5496845c699312ab20d6fb6aaa36ec6074006cac36d96c098906c650f121c" Oct 30 22:29:02.675442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.674176 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" Oct 30 22:29:02.751720 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.751672 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:29:02.751958 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.751916 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:29:02.861185 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.860089 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj"] Oct 30 22:29:02.885794 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:02.885761 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj"] Oct 30 22:29:03.573530 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:03.573495 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" path="/var/lib/kubelet/pods/3184ce2a-e642-4ac6-9bc8-fe1d65caec2f/volumes" Oct 30 22:29:03.663883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:03.663840 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:29:03.664176 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:03.664143 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:29:03.754580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:03.754540 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 22:29:03.754889 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:03.754854 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 22:29:04.651648 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:04.651598 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 22:29:04.652298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:04.652246 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 22:29:04.825673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:04.824527 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-f7c488-85f59cb498-jd6jj" podUID="3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.97:8443/health\": context deadline exceeded" Oct 30 22:29:05.370638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:05.370600 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:29:05.371721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:05.371702 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:29:05.372582 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:05.372564 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:29:06.574850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:06.574818 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:29:06.691586 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:06.691546 5139 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 22:29:06.691749 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:06.691613 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 22:29:07.163573 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:07.163545 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:29:07.163993 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:07.163971 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:29:07.164641 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:07.164621 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:29:17.037219 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.037182 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl"] Oct 30 22:29:17.037955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.037934 5139 cpu_manager.go:414] "containerMap: removing container" podUID="3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" containerName="maas-api" Oct 30 22:29:17.038054 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.037959 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" containerName="maas-api" Oct 30 22:29:17.038157 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.038104 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="3184ce2a-e642-4ac6-9bc8-fe1d65caec2f" containerName="maas-api" Oct 30 22:29:17.052816 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.052778 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.094427 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.094386 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl"] Oct 30 22:29:17.138536 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.138497 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6fc60c\"" Oct 30 22:29:17.138714 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.138578 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-6fc60c-dockercfg-qqlnp\"" Oct 30 22:29:17.174813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.174778 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93d2308-725f-402f-845e-041f586f08a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl\" (UID: \"f93d2308-725f-402f-845e-041f586f08a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.174974 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.174817 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lfts\" (UniqueName: \"kubernetes.io/projected/f93d2308-725f-402f-845e-041f586f08a6-kube-api-access-8lfts\") pod \"payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl\" (UID: \"f93d2308-725f-402f-845e-041f586f08a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.276571 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.276542 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93d2308-725f-402f-845e-041f586f08a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl\" (UID: \"f93d2308-725f-402f-845e-041f586f08a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.276732 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.276577 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lfts\" (UniqueName: \"kubernetes.io/projected/f93d2308-725f-402f-845e-041f586f08a6-kube-api-access-8lfts\") pod \"payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl\" (UID: \"f93d2308-725f-402f-845e-041f586f08a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.277272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.277251 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93d2308-725f-402f-845e-041f586f08a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl\" (UID: \"f93d2308-725f-402f-845e-041f586f08a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.334209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.334159 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lfts\" (UniqueName: \"kubernetes.io/projected/f93d2308-725f-402f-845e-041f586f08a6-kube-api-access-8lfts\") pod \"payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl\" (UID: \"f93d2308-725f-402f-845e-041f586f08a6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.375760 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.375741 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.746035 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.746006 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl"] Oct 30 22:29:17.754157 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:29:17.754125 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93d2308_725f_402f_845e_041f586f08a6.slice/crio-b6545efef3c84878db10c03a9aca8f367550650eea265275510b4dd896b3f254 WatchSource:0}: Error finding container b6545efef3c84878db10c03a9aca8f367550650eea265275510b4dd896b3f254: Status 404 returned error can't find the container with id b6545efef3c84878db10c03a9aca8f367550650eea265275510b4dd896b3f254 Oct 30 22:29:17.886128 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.886088 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" event={"ID":"f93d2308-725f-402f-845e-041f586f08a6","Type":"ContainerStarted","Data":"6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39"} Oct 30 22:29:17.886316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.886143 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" event={"ID":"f93d2308-725f-402f-845e-041f586f08a6","Type":"ContainerStarted","Data":"b6545efef3c84878db10c03a9aca8f367550650eea265275510b4dd896b3f254"} Oct 30 22:29:17.886316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.886175 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:17.996270 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:17.996176 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" podStartSLOduration=1.996164338 podStartE2EDuration="1.996164338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:17.994887881 +0000 UTC m=+3066.586173367" watchObservedRunningTime="2026-10-30 22:29:17.996164338 +0000 UTC m=+3066.587449824" Oct 30 22:29:19.804988 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:29:19.804944 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:29:25.332829 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.332746 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr"] Oct 30 22:29:25.337929 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.337912 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.390546 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.390522 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 22:29:25.390676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.390525 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 22:29:25.436369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.436329 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 22:29:25.436369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.436329 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-66p9x\"" Oct 30 22:29:25.438908 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.438881 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr"] Oct 30 22:29:25.482286 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.482258 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.482434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.482291 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.482434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.482326 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.482519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.482479 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8gqp\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-kube-api-access-p8gqp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.482554 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.482522 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.482554 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.482539 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.482620 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.482575 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.584301 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.584235 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.584301 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.584270 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.584540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.584310 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.584540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.584361 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8gqp\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-kube-api-access-p8gqp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.584540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.584500 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.584707 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.584543 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.584707 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.584568 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.585163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.585134 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.586770 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.586751 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.587242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.587221 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.587331 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.587289 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.587393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.587371 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.644071 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.644047 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8gqp\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-kube-api-access-p8gqp\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.645057 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.645035 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-lf2mr\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.649833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.649816 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:25.701176 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.701146 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt"] Oct 30 22:29:25.707898 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.707876 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.786600 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.786572 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.786761 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.786623 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9v27\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-kube-api-access-q9v27\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.786761 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.786747 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.786845 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.786832 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.786883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.786860 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.786980 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.786964 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.787020 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.786994 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fced4d-375e-4fa0-8d33-21329f3e7082-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.787105 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.787089 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.787151 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.787126 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.805611 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.805572 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt"] Oct 30 22:29:25.888142 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888060 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888145 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888174 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fced4d-375e-4fa0-8d33-21329f3e7082-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888250 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888486 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888291 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888486 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888320 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888486 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888369 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9v27\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-kube-api-access-q9v27\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888486 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888457 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888710 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888556 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888710 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888657 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888710 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888665 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888725 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.888953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888933 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fced4d-375e-4fa0-8d33-21329f3e7082-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.889006 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.888933 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.890718 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.890699 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.890878 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.890861 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.959224 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.959197 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9v27\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-kube-api-access-q9v27\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:25.959791 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:25.959766 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:26.030020 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.029993 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:26.082633 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.082073 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr"] Oct 30 22:29:26.104472 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.095066 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:29:26.104472 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.095371 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:29:26.408864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.408781 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt"] Oct 30 22:29:26.414161 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:29:26.414123 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fced4d_375e_4fa0_8d33_21329f3e7082.slice/crio-59443242768a047baf86dd3304e8450ae261f75b25bde3e4ae08d8172866f0ff WatchSource:0}: Error finding container 59443242768a047baf86dd3304e8450ae261f75b25bde3e4ae08d8172866f0ff: Status 404 returned error can't find the container with id 59443242768a047baf86dd3304e8450ae261f75b25bde3e4ae08d8172866f0ff Oct 30 22:29:26.417079 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.417045 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:29:26.417166 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.417113 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:29:26.417166 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.417140 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:29:26.936339 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.936303 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" event={"ID":"d9bf96bc-6f37-4a91-aa39-4a988b797bd8","Type":"ContainerStarted","Data":"8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987"} Oct 30 22:29:26.936560 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.936368 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:26.936560 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.936390 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" event={"ID":"d9bf96bc-6f37-4a91-aa39-4a988b797bd8","Type":"ContainerStarted","Data":"d59b075708cf95b22aed27db9947fb24a25b1e891bbf3181d3996b290606161d"} Oct 30 22:29:26.937987 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.937962 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" event={"ID":"c0fced4d-375e-4fa0-8d33-21329f3e7082","Type":"ContainerStarted","Data":"9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316"} Oct 30 22:29:26.938117 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:26.938007 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" event={"ID":"c0fced4d-375e-4fa0-8d33-21329f3e7082","Type":"ContainerStarted","Data":"59443242768a047baf86dd3304e8450ae261f75b25bde3e4ae08d8172866f0ff"} Oct 30 22:29:27.030398 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:27.030372 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:27.031971 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:27.031945 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:29:27.032076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:27.031992 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:29:27.052017 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:27.051976 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" podStartSLOduration=2.051963127 podStartE2EDuration="2.051963127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:27.050356393 +0000 UTC m=+3075.641641884" watchObservedRunningTime="2026-10-30 22:29:27.051963127 +0000 UTC m=+3075.643248614" Oct 30 22:29:27.175439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:27.175110 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podStartSLOduration=2.175085019 podStartE2EDuration="2.175085019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:27.172909835 +0000 UTC m=+3075.764195395" watchObservedRunningTime="2026-10-30 22:29:27.175085019 +0000 UTC m=+3075.766370507" Oct 30 22:29:27.947355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:27.947319 5139 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-lf2mr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:29:27.947756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:27.947393 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" podUID="d9bf96bc-6f37-4a91-aa39-4a988b797bd8" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:29:28.031172 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:28.031136 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:29:28.031353 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:28.031210 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:29:28.894207 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:28.894170 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:29:29.030624 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:29.030589 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:29:29.031011 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:29.030651 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:29:30.030741 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:30.030699 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:29:30.031500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:30.030770 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:29:30.946240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:30.946211 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:29:31.030718 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:31.030679 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:29:31.030889 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:31.030735 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:29:32.037846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:32.037808 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:29:32.038397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:32.038367 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:29:33.030858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:33.030820 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body= Oct 30 22:29:33.031047 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:33.030875 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" Oct 30 22:29:34.070813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:34.070781 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:34.071313 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:34.071173 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:34.071992 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:34.071968 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:29:49.807921 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:29:49.807874 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:29:54.092462 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:54.092426 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k"] Oct 30 22:29:54.092926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:54.092634 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" containerID="cri-o://e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a" gracePeriod=30 Oct 30 22:29:58.672188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:58.672149 5139 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:29:58.672776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:58.672219 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:29:59.357108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.357047 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:29:59.454976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.454946 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-credential-socket\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.454976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.454982 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-data\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.455246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455037 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-podinfo\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.455246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455134 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-socket\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.455246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455160 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9685434f-14f2-4ab9-b01a-e90458d910e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9685434f-14f2-4ab9-b01a-e90458d910e3-istiod-ca-cert\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.455446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455268 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-credential-socket" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:59.455446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455279 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-kube-api-access-hx6v6\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-kube-api-access-hx6v6\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.455446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455383 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-token\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-token\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.455446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455393 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-data" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:59.455653 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455543 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-socket" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:59.455653 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455629 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-envoy\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.455823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455675 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-certs\") pod \"9685434f-14f2-4ab9-b01a-e90458d910e3\" (UID: \"9685434f-14f2-4ab9-b01a-e90458d910e3\") " Oct 30 22:29:59.455823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455784 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9685434f-14f2-4ab9-b01a-e90458d910e3-istiod-ca-cert" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:29:59.455982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.455959 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-certs" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:59.456372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.456346 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:59.456372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.456373 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9685434f-14f2-4ab9-b01a-e90458d910e3-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:59.456578 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.456386 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:59.456578 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.456401 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:59.456578 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.456431 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:59.458439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.458382 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-envoy" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:29:59.458564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.458402 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-token" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:59.458564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.458532 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-kube-api-access-hx6v6" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "kube-api-access-hx6v6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:29:59.458564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.458531 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-podinfo" pod "9685434f-14f2-4ab9-b01a-e90458d910e3" (UID: "9685434f-14f2-4ab9-b01a-e90458d910e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:29:59.558651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.558375 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx6v6\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-kube-api-access-hx6v6\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:59.558651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.558409 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:59.558651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.558457 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:29:59.558651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:29:59.558471 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9685434f-14f2-4ab9-b01a-e90458d910e3-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:00.152657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.152623 5139 generic.go:356] "Generic (PLEG): container finished" podID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerID="e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a" exitCode=0 Oct 30 22:30:00.153067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.152692 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" Oct 30 22:30:00.153067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.152705 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" event={"ID":"9685434f-14f2-4ab9-b01a-e90458d910e3","Type":"ContainerDied","Data":"e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a"} Oct 30 22:30:00.153067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.152760 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k" event={"ID":"9685434f-14f2-4ab9-b01a-e90458d910e3","Type":"ContainerDied","Data":"c06c9d8be1955bd7fea0ba0d5739179750ceedc347ec1e09f9fccd649b82ce7e"} Oct 30 22:30:00.153067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.152792 5139 scope.go:122] "RemoveContainer" containerID="e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a" Oct 30 22:30:00.161941 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.161921 5139 scope.go:122] "RemoveContainer" containerID="e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a" Oct 30 22:30:00.162180 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:00.162160 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a\": container with ID starting with e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a not found: ID does not exist" containerID="e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a" Oct 30 22:30:00.162256 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.162187 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a"} err="failed to get container status \"e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a\": rpc error: code = NotFound desc = could not find container \"e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a\": container with ID starting with e10e4f4a957930f14989344c032830ce5b5994ef036e086c8e39761eac3be40a not found: ID does not exist" Oct 30 22:30:00.263975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.263934 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b"] Oct 30 22:30:00.264569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.264549 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" Oct 30 22:30:00.264628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.264571 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" Oct 30 22:30:00.264664 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.264657 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" containerName="istio-proxy" Oct 30 22:30:00.269315 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.269292 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7"] Oct 30 22:30:00.269446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.269406 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:00.272850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.272831 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt"] Oct 30 22:30:00.272958 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.272943 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:00.276329 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.276308 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:00.298884 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.298864 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b"] Oct 30 22:30:00.316404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.316377 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7"] Oct 30 22:30:00.316555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.316449 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt"] Oct 30 22:30:00.354504 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.354479 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\"" Oct 30 22:30:00.366948 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.366915 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zfp\" (UniqueName: \"kubernetes.io/projected/c65c60a4-d560-4940-acec-296ae534cbc5-kube-api-access-x2zfp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7\" (UID: \"c65c60a4-d560-4940-acec-296ae534cbc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:00.367092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.366988 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2112b6a-4428-483c-802d-11bde8df48c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt\" (UID: \"c2112b6a-4428-483c-802d-11bde8df48c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:00.367171 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.367082 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwk86\" (UniqueName: \"kubernetes.io/projected/c2112b6a-4428-483c-802d-11bde8df48c2-kube-api-access-cwk86\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt\" (UID: \"c2112b6a-4428-483c-802d-11bde8df48c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:00.367171 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.367136 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mw86\" (UniqueName: \"kubernetes.io/projected/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-kube-api-access-6mw86\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b\" (UID: \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:00.367359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.367337 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65c60a4-d560-4940-acec-296ae534cbc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7\" (UID: \"c65c60a4-d560-4940-acec-296ae534cbc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:00.367453 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.367435 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b\" (UID: \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:00.377425 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.377395 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k"] Oct 30 22:30:00.383038 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.383018 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-ndd7k"] Oct 30 22:30:00.468734 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.468654 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b\" (UID: \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:00.468734 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.468718 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zfp\" (UniqueName: \"kubernetes.io/projected/c65c60a4-d560-4940-acec-296ae534cbc5-kube-api-access-x2zfp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7\" (UID: \"c65c60a4-d560-4940-acec-296ae534cbc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:00.468962 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.468754 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2112b6a-4428-483c-802d-11bde8df48c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt\" (UID: \"c2112b6a-4428-483c-802d-11bde8df48c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:00.468962 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.468782 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwk86\" (UniqueName: \"kubernetes.io/projected/c2112b6a-4428-483c-802d-11bde8df48c2-kube-api-access-cwk86\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt\" (UID: \"c2112b6a-4428-483c-802d-11bde8df48c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:00.468962 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.468811 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mw86\" (UniqueName: \"kubernetes.io/projected/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-kube-api-access-6mw86\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b\" (UID: \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:00.468962 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.468904 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65c60a4-d560-4940-acec-296ae534cbc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7\" (UID: \"c65c60a4-d560-4940-acec-296ae534cbc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:00.469506 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.469481 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b\" (UID: \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:00.469611 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.469483 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2112b6a-4428-483c-802d-11bde8df48c2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt\" (UID: \"c2112b6a-4428-483c-802d-11bde8df48c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:00.469611 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.469533 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65c60a4-d560-4940-acec-296ae534cbc5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7\" (UID: \"c65c60a4-d560-4940-acec-296ae534cbc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:00.532438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.532399 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mw86\" (UniqueName: \"kubernetes.io/projected/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-kube-api-access-6mw86\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b\" (UID: \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:00.532565 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.532406 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwk86\" (UniqueName: \"kubernetes.io/projected/c2112b6a-4428-483c-802d-11bde8df48c2-kube-api-access-cwk86\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt\" (UID: \"c2112b6a-4428-483c-802d-11bde8df48c2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:00.532565 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.532482 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zfp\" (UniqueName: \"kubernetes.io/projected/c65c60a4-d560-4940-acec-296ae534cbc5-kube-api-access-x2zfp\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7\" (UID: \"c65c60a4-d560-4940-acec-296ae534cbc5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:00.584026 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.584003 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:00.595019 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.594991 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:00.607683 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.607664 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:00.969403 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:00.969369 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b"] Oct 30 22:30:00.975669 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:30:00.975635 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4a6d6f_9d08_4ba8_a5ee_cace11938f04.slice/crio-96e5675c91488c1c0a9eba1ff16d098a037f9907a803f01b88fd5f58985606e3 WatchSource:0}: Error finding container 96e5675c91488c1c0a9eba1ff16d098a037f9907a803f01b88fd5f58985606e3: Status 404 returned error can't find the container with id 96e5675c91488c1c0a9eba1ff16d098a037f9907a803f01b88fd5f58985606e3 Oct 30 22:30:01.019692 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:01.019665 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7"] Oct 30 22:30:01.023580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:01.023560 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt"] Oct 30 22:30:01.026252 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:30:01.026231 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65c60a4_d560_4940_acec_296ae534cbc5.slice/crio-0f09e098c85449e70e9fd9af3e9a13c10361fb7bb6104ebb58f5c6e8e82372b0 WatchSource:0}: Error finding container 0f09e098c85449e70e9fd9af3e9a13c10361fb7bb6104ebb58f5c6e8e82372b0: Status 404 returned error can't find the container with id 0f09e098c85449e70e9fd9af3e9a13c10361fb7bb6104ebb58f5c6e8e82372b0 Oct 30 22:30:01.029257 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:30:01.029235 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2112b6a_4428_483c_802d_11bde8df48c2.slice/crio-309ced5226a8df786d3038c44a4edcb867f4759790c3ef7ea3e7db83d5ac0a53 WatchSource:0}: Error finding container 309ced5226a8df786d3038c44a4edcb867f4759790c3ef7ea3e7db83d5ac0a53: Status 404 returned error can't find the container with id 309ced5226a8df786d3038c44a4edcb867f4759790c3ef7ea3e7db83d5ac0a53 Oct 30 22:30:01.167062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:01.167021 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" event={"ID":"c2112b6a-4428-483c-802d-11bde8df48c2","Type":"ContainerStarted","Data":"309ced5226a8df786d3038c44a4edcb867f4759790c3ef7ea3e7db83d5ac0a53"} Oct 30 22:30:01.168518 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:01.168490 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" event={"ID":"c65c60a4-d560-4940-acec-296ae534cbc5","Type":"ContainerStarted","Data":"0f09e098c85449e70e9fd9af3e9a13c10361fb7bb6104ebb58f5c6e8e82372b0"} Oct 30 22:30:01.169868 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:01.169843 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" event={"ID":"db4a6d6f-9d08-4ba8-a5ee-cace11938f04","Type":"ContainerStarted","Data":"96e5675c91488c1c0a9eba1ff16d098a037f9907a803f01b88fd5f58985606e3"} Oct 30 22:30:01.547957 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:01.547865 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9685434f-14f2-4ab9-b01a-e90458d910e3" path="/var/lib/kubelet/pods/9685434f-14f2-4ab9-b01a-e90458d910e3/volumes" Oct 30 22:30:02.177122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.177035 5139 generic.go:356] "Generic (PLEG): container finished" podID="c65c60a4-d560-4940-acec-296ae534cbc5" containerID="5720e7ced35ac6e47a3b44745382e72e653602a0540c7f8c0b72f0d6afdd2738" exitCode=7 Oct 30 22:30:02.177587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.177117 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" event={"ID":"c65c60a4-d560-4940-acec-296ae534cbc5","Type":"ContainerDied","Data":"5720e7ced35ac6e47a3b44745382e72e653602a0540c7f8c0b72f0d6afdd2738"} Oct 30 22:30:02.177587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.177430 5139 scope.go:122] "RemoveContainer" containerID="5720e7ced35ac6e47a3b44745382e72e653602a0540c7f8c0b72f0d6afdd2738" Oct 30 22:30:02.178858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.178833 5139 generic.go:356] "Generic (PLEG): container finished" podID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerID="91068d90f511ac9c722525e35bedd6d21d0a2ca9c6e14b5c399b9c25432d0b39" exitCode=60 Oct 30 22:30:02.178959 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.178932 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" event={"ID":"db4a6d6f-9d08-4ba8-a5ee-cace11938f04","Type":"ContainerDied","Data":"91068d90f511ac9c722525e35bedd6d21d0a2ca9c6e14b5c399b9c25432d0b39"} Oct 30 22:30:02.179239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.179214 5139 scope.go:122] "RemoveContainer" containerID="91068d90f511ac9c722525e35bedd6d21d0a2ca9c6e14b5c399b9c25432d0b39" Oct 30 22:30:02.180778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.180627 5139 generic.go:356] "Generic (PLEG): container finished" podID="c2112b6a-4428-483c-802d-11bde8df48c2" containerID="71060d5dd9bcc776b8b680543be79dba75b76d6961f6f449e9cbda85383881e3" exitCode=60 Oct 30 22:30:02.180778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.180709 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" event={"ID":"c2112b6a-4428-483c-802d-11bde8df48c2","Type":"ContainerDied","Data":"71060d5dd9bcc776b8b680543be79dba75b76d6961f6f449e9cbda85383881e3"} Oct 30 22:30:02.180907 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:02.180892 5139 scope.go:122] "RemoveContainer" containerID="71060d5dd9bcc776b8b680543be79dba75b76d6961f6f449e9cbda85383881e3" Oct 30 22:30:03.188574 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.188546 5139 generic.go:356] "Generic (PLEG): container finished" podID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerID="cf6e60e661a9d0baec4b274f74f59de6dd90fbd2f7751dfdf9d635c9006c02d6" exitCode=60 Oct 30 22:30:03.189004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.188625 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" event={"ID":"db4a6d6f-9d08-4ba8-a5ee-cace11938f04","Type":"ContainerDied","Data":"cf6e60e661a9d0baec4b274f74f59de6dd90fbd2f7751dfdf9d635c9006c02d6"} Oct 30 22:30:03.189004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.188697 5139 scope.go:122] "RemoveContainer" containerID="91068d90f511ac9c722525e35bedd6d21d0a2ca9c6e14b5c399b9c25432d0b39" Oct 30 22:30:03.189004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.188980 5139 scope.go:122] "RemoveContainer" containerID="cf6e60e661a9d0baec4b274f74f59de6dd90fbd2f7751dfdf9d635c9006c02d6" Oct 30 22:30:03.189339 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:03.189312 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b_odh-ai-gateway-infra(db4a6d6f-9d08-4ba8-a5ee-cace11938f04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" Oct 30 22:30:03.190926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.190905 5139 generic.go:356] "Generic (PLEG): container finished" podID="c2112b6a-4428-483c-802d-11bde8df48c2" containerID="a61e3969adad401539e3e1e104865af486943a835276cd118d999917e6750d8c" exitCode=60 Oct 30 22:30:03.191051 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.190941 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" event={"ID":"c2112b6a-4428-483c-802d-11bde8df48c2","Type":"ContainerDied","Data":"a61e3969adad401539e3e1e104865af486943a835276cd118d999917e6750d8c"} Oct 30 22:30:03.191272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.191255 5139 scope.go:122] "RemoveContainer" containerID="a61e3969adad401539e3e1e104865af486943a835276cd118d999917e6750d8c" Oct 30 22:30:03.191584 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:03.191555 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt_odh-ai-gateway-infra(c2112b6a-4428-483c-802d-11bde8df48c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" Oct 30 22:30:03.192880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.192857 5139 generic.go:356] "Generic (PLEG): container finished" podID="c65c60a4-d560-4940-acec-296ae534cbc5" containerID="963211bfb8ae810a9b3a542963287ee791b1e382e59d9c37191a83a2de8fc4af" exitCode=7 Oct 30 22:30:03.192969 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.192891 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" event={"ID":"c65c60a4-d560-4940-acec-296ae534cbc5","Type":"ContainerDied","Data":"963211bfb8ae810a9b3a542963287ee791b1e382e59d9c37191a83a2de8fc4af"} Oct 30 22:30:03.193137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.193124 5139 scope.go:122] "RemoveContainer" containerID="963211bfb8ae810a9b3a542963287ee791b1e382e59d9c37191a83a2de8fc4af" Oct 30 22:30:03.193370 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:03.193351 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7_odh-ai-gateway-infra(c65c60a4-d560-4940-acec-296ae534cbc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" Oct 30 22:30:03.321938 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.321914 5139 scope.go:122] "RemoveContainer" containerID="71060d5dd9bcc776b8b680543be79dba75b76d6961f6f449e9cbda85383881e3" Oct 30 22:30:03.334519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:03.334500 5139 scope.go:122] "RemoveContainer" containerID="5720e7ced35ac6e47a3b44745382e72e653602a0540c7f8c0b72f0d6afdd2738" Oct 30 22:30:04.199915 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:04.199887 5139 scope.go:122] "RemoveContainer" containerID="963211bfb8ae810a9b3a542963287ee791b1e382e59d9c37191a83a2de8fc4af" Oct 30 22:30:04.200353 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:04.200238 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7_odh-ai-gateway-infra(c65c60a4-d560-4940-acec-296ae534cbc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" Oct 30 22:30:04.201390 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:04.201370 5139 scope.go:122] "RemoveContainer" containerID="cf6e60e661a9d0baec4b274f74f59de6dd90fbd2f7751dfdf9d635c9006c02d6" Oct 30 22:30:04.202245 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:04.202204 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b_odh-ai-gateway-infra(db4a6d6f-9d08-4ba8-a5ee-cace11938f04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" Oct 30 22:30:04.210312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:04.210289 5139 scope.go:122] "RemoveContainer" containerID="a61e3969adad401539e3e1e104865af486943a835276cd118d999917e6750d8c" Oct 30 22:30:04.210701 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:04.210676 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt_odh-ai-gateway-infra(c2112b6a-4428-483c-802d-11bde8df48c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" Oct 30 22:30:15.536011 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:15.535969 5139 scope.go:122] "RemoveContainer" containerID="963211bfb8ae810a9b3a542963287ee791b1e382e59d9c37191a83a2de8fc4af" Oct 30 22:30:16.280734 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:16.280700 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" event={"ID":"c65c60a4-d560-4940-acec-296ae534cbc5","Type":"ContainerStarted","Data":"d31e975d19ca7b9ab83b858623570e3190fa5be9226dc261bc29907e10733055"} Oct 30 22:30:16.392888 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:16.392830 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" podStartSLOduration=16.392814813 podStartE2EDuration="16.392814813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:16.389849406 +0000 UTC m=+3124.981134892" watchObservedRunningTime="2026-10-30 22:30:16.392814813 +0000 UTC m=+3124.984100300" Oct 30 22:30:16.535818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:16.535745 5139 scope.go:122] "RemoveContainer" containerID="a61e3969adad401539e3e1e104865af486943a835276cd118d999917e6750d8c" Oct 30 22:30:16.535942 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:16.535902 5139 scope.go:122] "RemoveContainer" containerID="cf6e60e661a9d0baec4b274f74f59de6dd90fbd2f7751dfdf9d635c9006c02d6" Oct 30 22:30:17.291104 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.291066 5139 generic.go:356] "Generic (PLEG): container finished" podID="c65c60a4-d560-4940-acec-296ae534cbc5" containerID="d31e975d19ca7b9ab83b858623570e3190fa5be9226dc261bc29907e10733055" exitCode=60 Oct 30 22:30:17.291672 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.291107 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" event={"ID":"c65c60a4-d560-4940-acec-296ae534cbc5","Type":"ContainerDied","Data":"d31e975d19ca7b9ab83b858623570e3190fa5be9226dc261bc29907e10733055"} Oct 30 22:30:17.291672 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.291185 5139 scope.go:122] "RemoveContainer" containerID="963211bfb8ae810a9b3a542963287ee791b1e382e59d9c37191a83a2de8fc4af" Oct 30 22:30:17.291672 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.291614 5139 scope.go:122] "RemoveContainer" containerID="d31e975d19ca7b9ab83b858623570e3190fa5be9226dc261bc29907e10733055" Oct 30 22:30:17.291943 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:17.291918 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7_odh-ai-gateway-infra(c65c60a4-d560-4940-acec-296ae534cbc5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" Oct 30 22:30:17.294129 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.294103 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" event={"ID":"db4a6d6f-9d08-4ba8-a5ee-cace11938f04","Type":"ContainerStarted","Data":"25d8acffd4f03ade09600a4be8f897f97e70536c33490a68ba50138c69208b88"} Oct 30 22:30:17.297034 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.297008 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" event={"ID":"c2112b6a-4428-483c-802d-11bde8df48c2","Type":"ContainerStarted","Data":"050770b6b37b386cfef0b4b47ed932ef5d450487d6eefbeeb27960bc0e36fae1"} Oct 30 22:30:17.405101 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.405073 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7"] Oct 30 22:30:17.539942 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.539873 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" podStartSLOduration=17.5398545 podStartE2EDuration="17.5398545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:17.538541282 +0000 UTC m=+3126.129826770" watchObservedRunningTime="2026-10-30 22:30:17.5398545 +0000 UTC m=+3126.131139986" Oct 30 22:30:17.649311 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:17.649215 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" podStartSLOduration=17.649199982 podStartE2EDuration="17.649199982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:17.647983871 +0000 UTC m=+3126.239269355" watchObservedRunningTime="2026-10-30 22:30:17.649199982 +0000 UTC m=+3126.240485469" Oct 30 22:30:18.308186 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.308149 5139 generic.go:356] "Generic (PLEG): container finished" podID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerID="25d8acffd4f03ade09600a4be8f897f97e70536c33490a68ba50138c69208b88" exitCode=60 Oct 30 22:30:18.308675 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.308221 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" event={"ID":"db4a6d6f-9d08-4ba8-a5ee-cace11938f04","Type":"ContainerDied","Data":"25d8acffd4f03ade09600a4be8f897f97e70536c33490a68ba50138c69208b88"} Oct 30 22:30:18.308675 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.308296 5139 scope.go:122] "RemoveContainer" containerID="cf6e60e661a9d0baec4b274f74f59de6dd90fbd2f7751dfdf9d635c9006c02d6" Oct 30 22:30:18.308791 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.308725 5139 scope.go:122] "RemoveContainer" containerID="25d8acffd4f03ade09600a4be8f897f97e70536c33490a68ba50138c69208b88" Oct 30 22:30:18.309122 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:18.309094 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b_odh-ai-gateway-infra(db4a6d6f-9d08-4ba8-a5ee-cace11938f04)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" Oct 30 22:30:18.311647 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.310832 5139 generic.go:356] "Generic (PLEG): container finished" podID="c2112b6a-4428-483c-802d-11bde8df48c2" containerID="050770b6b37b386cfef0b4b47ed932ef5d450487d6eefbeeb27960bc0e36fae1" exitCode=60 Oct 30 22:30:18.311647 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.310875 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" event={"ID":"c2112b6a-4428-483c-802d-11bde8df48c2","Type":"ContainerDied","Data":"050770b6b37b386cfef0b4b47ed932ef5d450487d6eefbeeb27960bc0e36fae1"} Oct 30 22:30:18.311647 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.311263 5139 scope.go:122] "RemoveContainer" containerID="050770b6b37b386cfef0b4b47ed932ef5d450487d6eefbeeb27960bc0e36fae1" Oct 30 22:30:18.311647 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:18.311587 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt_odh-ai-gateway-infra(c2112b6a-4428-483c-802d-11bde8df48c2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" Oct 30 22:30:18.321228 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.321205 5139 scope.go:122] "RemoveContainer" containerID="a61e3969adad401539e3e1e104865af486943a835276cd118d999917e6750d8c" Oct 30 22:30:18.460015 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.459988 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:18.551100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.551069 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b"] Oct 30 22:30:18.597766 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.597684 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c65c60a4-d560-4940-acec-296ae534cbc5-kube-api-access-x2zfp\" (UniqueName: \"kubernetes.io/projected/c65c60a4-d560-4940-acec-296ae534cbc5-kube-api-access-x2zfp\") pod \"c65c60a4-d560-4940-acec-296ae534cbc5\" (UID: \"c65c60a4-d560-4940-acec-296ae534cbc5\") " Oct 30 22:30:18.597939 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.597873 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c65c60a4-d560-4940-acec-296ae534cbc5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65c60a4-d560-4940-acec-296ae534cbc5-maas-api-service-ca\") pod \"c65c60a4-d560-4940-acec-296ae534cbc5\" (UID: \"c65c60a4-d560-4940-acec-296ae534cbc5\") " Oct 30 22:30:18.598394 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.598367 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c65c60a4-d560-4940-acec-296ae534cbc5-maas-api-service-ca" pod "c65c60a4-d560-4940-acec-296ae534cbc5" (UID: "c65c60a4-d560-4940-acec-296ae534cbc5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:18.600367 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.600334 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c65c60a4-d560-4940-acec-296ae534cbc5-kube-api-access-x2zfp" pod "c65c60a4-d560-4940-acec-296ae534cbc5" (UID: "c65c60a4-d560-4940-acec-296ae534cbc5"). InnerVolumeSpecName "kube-api-access-x2zfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:18.658976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.658944 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt"] Oct 30 22:30:18.699211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.699172 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2zfp\" (UniqueName: \"kubernetes.io/projected/c65c60a4-d560-4940-acec-296ae534cbc5-kube-api-access-x2zfp\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:18.699211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:18.699204 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c65c60a4-d560-4940-acec-296ae534cbc5-maas-api-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.320209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.320183 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" Oct 30 22:30:19.320209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.320183 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7" event={"ID":"c65c60a4-d560-4940-acec-296ae534cbc5","Type":"ContainerDied","Data":"0f09e098c85449e70e9fd9af3e9a13c10361fb7bb6104ebb58f5c6e8e82372b0"} Oct 30 22:30:19.320677 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.320289 5139 scope.go:122] "RemoveContainer" containerID="d31e975d19ca7b9ab83b858623570e3190fa5be9226dc261bc29907e10733055" Oct 30 22:30:19.488625 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.488598 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:19.520823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.520796 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:19.610653 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.610568 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2112b6a-4428-483c-802d-11bde8df48c2-kube-api-access-cwk86\" (UniqueName: \"kubernetes.io/projected/c2112b6a-4428-483c-802d-11bde8df48c2-kube-api-access-cwk86\") pod \"c2112b6a-4428-483c-802d-11bde8df48c2\" (UID: \"c2112b6a-4428-483c-802d-11bde8df48c2\") " Oct 30 22:30:19.610653 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.610654 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2112b6a-4428-483c-802d-11bde8df48c2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2112b6a-4428-483c-802d-11bde8df48c2-maas-api-service-ca\") pod \"c2112b6a-4428-483c-802d-11bde8df48c2\" (UID: \"c2112b6a-4428-483c-802d-11bde8df48c2\") " Oct 30 22:30:19.610892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.610682 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-kube-api-access-6mw86\" (UniqueName: \"kubernetes.io/projected/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-kube-api-access-6mw86\") pod \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\" (UID: \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\") " Oct 30 22:30:19.610892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.610736 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-maas-api-service-ca\") pod \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\" (UID: \"db4a6d6f-9d08-4ba8-a5ee-cace11938f04\") " Oct 30 22:30:19.611039 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.611015 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2112b6a-4428-483c-802d-11bde8df48c2-maas-api-service-ca" pod "c2112b6a-4428-483c-802d-11bde8df48c2" (UID: "c2112b6a-4428-483c-802d-11bde8df48c2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.611197 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.611167 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-maas-api-service-ca" pod "db4a6d6f-9d08-4ba8-a5ee-cace11938f04" (UID: "db4a6d6f-9d08-4ba8-a5ee-cace11938f04"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:30:19.611386 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.611367 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2112b6a-4428-483c-802d-11bde8df48c2-maas-api-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.611475 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.611393 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-maas-api-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.613030 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.613003 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2112b6a-4428-483c-802d-11bde8df48c2-kube-api-access-cwk86" pod "c2112b6a-4428-483c-802d-11bde8df48c2" (UID: "c2112b6a-4428-483c-802d-11bde8df48c2"). InnerVolumeSpecName "kube-api-access-cwk86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.613030 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.613023 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-kube-api-access-6mw86" pod "db4a6d6f-9d08-4ba8-a5ee-cace11938f04" (UID: "db4a6d6f-9d08-4ba8-a5ee-cace11938f04"). InnerVolumeSpecName "kube-api-access-6mw86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:30:19.646159 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.646131 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7"] Oct 30 22:30:19.652721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.652696 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29889990-zbhg7"] Oct 30 22:30:19.712170 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.712135 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mw86\" (UniqueName: \"kubernetes.io/projected/db4a6d6f-9d08-4ba8-a5ee-cace11938f04-kube-api-access-6mw86\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.712170 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:19.712166 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwk86\" (UniqueName: \"kubernetes.io/projected/c2112b6a-4428-483c-802d-11bde8df48c2-kube-api-access-cwk86\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:30:19.806583 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:19.806544 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:30:20.329824 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.329790 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" Oct 30 22:30:20.330288 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.329821 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt" event={"ID":"c2112b6a-4428-483c-802d-11bde8df48c2","Type":"ContainerDied","Data":"309ced5226a8df786d3038c44a4edcb867f4759790c3ef7ea3e7db83d5ac0a53"} Oct 30 22:30:20.330288 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.329881 5139 scope.go:122] "RemoveContainer" containerID="050770b6b37b386cfef0b4b47ed932ef5d450487d6eefbeeb27960bc0e36fae1" Oct 30 22:30:20.332203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.332178 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" event={"ID":"db4a6d6f-9d08-4ba8-a5ee-cace11938f04","Type":"ContainerDied","Data":"96e5675c91488c1c0a9eba1ff16d098a037f9907a803f01b88fd5f58985606e3"} Oct 30 22:30:20.332203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.332200 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b" Oct 30 22:30:20.340400 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.340382 5139 scope.go:122] "RemoveContainer" containerID="25d8acffd4f03ade09600a4be8f897f97e70536c33490a68ba50138c69208b88" Oct 30 22:30:20.484865 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.484834 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b"] Oct 30 22:30:20.487239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.487209 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29889990-v2q2b"] Oct 30 22:30:20.591896 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.591785 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt"] Oct 30 22:30:20.594292 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:20.594258 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-43d592-29889990-77rxt"] Oct 30 22:30:21.552260 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:21.552222 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" path="/var/lib/kubelet/pods/c2112b6a-4428-483c-802d-11bde8df48c2/volumes" Oct 30 22:30:21.552806 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:21.552741 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" path="/var/lib/kubelet/pods/c65c60a4-d560-4940-acec-296ae534cbc5/volumes" Oct 30 22:30:21.553154 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:21.553134 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" path="/var/lib/kubelet/pods/db4a6d6f-9d08-4ba8-a5ee-cace11938f04/volumes" Oct 30 22:30:43.571299 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.571264 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps"] Oct 30 22:30:43.572046 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572023 5139 cpu_manager.go:414] "containerMap: removing container" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572048 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572061 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572068 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572077 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572084 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572118 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572125 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572140 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:30:43.572149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572148 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572157 5139 cpu_manager.go:414] "containerMap: removing container" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572164 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572307 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572323 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572361 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572371 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572387 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572398 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572406 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:30:43.572731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.572444 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:30:43.575982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.575962 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634208 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-credential-socket\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634262 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-envoy\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634351 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-token\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634379 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-certs\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634462 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-podinfo\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634509 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-socket\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634562 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da5aadf-467a-4499-ae82-f7e05e2bde19-istiod-ca-cert\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634613 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkn7\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-kube-api-access-nrkn7\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.634811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.634662 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-data\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.662268 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.659786 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-5bb0b1-openshift-default-dockercfg-ckxgl\"" Oct 30 22:30:43.685858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.685763 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps"] Oct 30 22:30:43.735943 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.735905 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkn7\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-kube-api-access-nrkn7\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.735963 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-data\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736030 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-credential-socket\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736054 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-envoy\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736117 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-token\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736144 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-certs\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736167 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-podinfo\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736554 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736431 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-socket\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736554 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736483 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-data\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736554 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736516 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da5aadf-467a-4499-ae82-f7e05e2bde19-istiod-ca-cert\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736673 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-certs\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736790 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736721 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-credential-socket\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.736790 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.736753 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-socket\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.737245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.737224 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da5aadf-467a-4499-ae82-f7e05e2bde19-istiod-ca-cert\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.738464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.738446 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-envoy\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.738622 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.738607 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-podinfo\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.794136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.794105 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-token\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.794136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.794131 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkn7\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-kube-api-access-nrkn7\") pod \"e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:43.899240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:43.892097 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:44.331612 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:44.331578 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps"] Oct 30 22:30:44.338157 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:30:44.338126 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da5aadf_467a_4499_ae82_f7e05e2bde19.slice/crio-e7c79f4fca5d397289eb4366db539e5ecb9a0688a1971c80bf8de3f021be182a WatchSource:0}: Error finding container e7c79f4fca5d397289eb4366db539e5ecb9a0688a1971c80bf8de3f021be182a: Status 404 returned error can't find the container with id e7c79f4fca5d397289eb4366db539e5ecb9a0688a1971c80bf8de3f021be182a Oct 30 22:30:44.340776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:44.340734 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:30:44.340885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:44.340827 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:30:44.340885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:44.340875 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:30:44.494148 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:44.494099 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" event={"ID":"8da5aadf-467a-4499-ae82-f7e05e2bde19","Type":"ContainerStarted","Data":"7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d"} Oct 30 22:30:44.494317 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:44.494179 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" event={"ID":"8da5aadf-467a-4499-ae82-f7e05e2bde19","Type":"ContainerStarted","Data":"e7c79f4fca5d397289eb4366db539e5ecb9a0688a1971c80bf8de3f021be182a"} Oct 30 22:30:44.642033 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:44.641909 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podStartSLOduration=1.641889658 podStartE2EDuration="1.641889658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:44.641604731 +0000 UTC m=+3153.232890217" watchObservedRunningTime="2026-10-30 22:30:44.641889658 +0000 UTC m=+3153.233175146" Oct 30 22:30:44.898141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:44.898052 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:45.903696 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:45.903663 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:46.508163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:46.508131 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:46.509127 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:30:46.509109 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:30:49.806921 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:30:49.806878 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:31:19.663634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.663596 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l"] Oct 30 22:31:19.665017 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.664990 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:31:19.665017 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.665014 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:31:19.665223 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.665032 5139 cpu_manager.go:414] "containerMap: removing container" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:31:19.665223 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.665040 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4a6d6f-9d08-4ba8-a5ee-cace11938f04" containerName="cleanup" Oct 30 22:31:19.665223 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.665091 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:31:19.665223 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.665099 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c65c60a4-d560-4940-acec-296ae534cbc5" containerName="cleanup" Oct 30 22:31:19.665455 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.665245 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2112b6a-4428-483c-802d-11bde8df48c2" containerName="cleanup" Oct 30 22:31:19.670445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.670402 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706328 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706297 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-data\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706328 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706332 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-certs\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706358 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-token\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706503 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-podinfo\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706532 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-credential-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706570 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-envoy\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706889 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706656 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5p6\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-kube-api-access-kk5p6\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706889 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706702 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79e9de-5f9e-497f-8892-1a7341d92672-istiod-ca-cert\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.706889 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.706760 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.721715 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.721684 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-355213-openshift-default-dockercfg-k56hc\"" Oct 30 22:31:19.772287 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.772231 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l"] Oct 30 22:31:19.807624 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807592 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-data\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.807814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807635 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-certs\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.807814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807659 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-token\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.807814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807699 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-podinfo\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.807814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807722 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-credential-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.807814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807768 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-envoy\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.808073 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807817 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5p6\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-kube-api-access-kk5p6\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.808073 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807875 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79e9de-5f9e-497f-8892-1a7341d92672-istiod-ca-cert\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.808073 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.807939 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.808242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.808066 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-data\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.808322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.808300 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.808387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.808354 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-credential-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.810802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.808647 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-certs\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.810802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.809501 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79e9de-5f9e-497f-8892-1a7341d92672-istiod-ca-cert\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.810962 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:31:19.810825 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:31:19.812814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.812789 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-podinfo\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.812959 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.812831 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-envoy\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.873860 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.873828 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-token\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.874015 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.873948 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5p6\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-kube-api-access-kk5p6\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-dpv7l\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:19.984917 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:19.984832 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:20.417198 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:20.417164 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l"] Oct 30 22:31:20.421576 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:31:20.421551 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c79e9de_5f9e_497f_8892_1a7341d92672.slice/crio-3c3a6eb850f887a7b1bbca947f38ca1fed4fb4a5664e71720fb554f647c06a31 WatchSource:0}: Error finding container 3c3a6eb850f887a7b1bbca947f38ca1fed4fb4a5664e71720fb554f647c06a31: Status 404 returned error can't find the container with id 3c3a6eb850f887a7b1bbca947f38ca1fed4fb4a5664e71720fb554f647c06a31 Oct 30 22:31:20.424111 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:20.424076 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:31:20.424194 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:20.424150 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:31:20.424194 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:20.424188 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:31:20.714575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:20.714470 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" event={"ID":"6c79e9de-5f9e-497f-8892-1a7341d92672","Type":"ContainerStarted","Data":"ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420"} Oct 30 22:31:20.714575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:20.714547 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" event={"ID":"6c79e9de-5f9e-497f-8892-1a7341d92672","Type":"ContainerStarted","Data":"3c3a6eb850f887a7b1bbca947f38ca1fed4fb4a5664e71720fb554f647c06a31"} Oct 30 22:31:20.832253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:20.832195 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" podStartSLOduration=1.832180527 podStartE2EDuration="1.832180527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:20.830570579 +0000 UTC m=+3189.421856065" watchObservedRunningTime="2026-10-30 22:31:20.832180527 +0000 UTC m=+3189.423466014" Oct 30 22:31:20.985430 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:20.985325 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:21.991546 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:21.991514 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:22.725988 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:22.725956 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:22.726845 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:22.726821 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:31:49.804092 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:31:49.804050 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:31:52.144814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.144784 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq"] Oct 30 22:31:52.149074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.149054 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.197944 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.197916 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-ae7a1f-serving-cert\"" Oct 30 22:31:52.202509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.202488 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgvjf\" (UniqueName: \"kubernetes.io/projected/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-kube-api-access-jgvjf\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.202658 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.202642 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.202709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.202686 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.202770 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.202758 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.248309 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.248282 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq"] Oct 30 22:31:52.304346 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.304311 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.304563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.304376 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.304563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.304459 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.304563 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:31:52.304497 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-serving-cert: secret "maas-api-e2e-shared-b-w3-ae7a1f-serving-cert" not found Oct 30 22:31:52.304563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.304557 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgvjf\" (UniqueName: \"kubernetes.io/projected/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-kube-api-access-jgvjf\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.304792 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:31:52.304574 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls podName:4c240dd4-fd65-4f9d-89d6-593d39ae0ebc nodeName:}" failed. No retries permitted until 2026-10-30 22:31:52.804551977 +0000 UTC m=+3221.395837465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls") pod "maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" (UID: "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc") : secret "maas-api-e2e-shared-b-w3-ae7a1f-serving-cert" not found Oct 30 22:31:52.305226 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.305196 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.306969 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.306946 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.362347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.362318 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgvjf\" (UniqueName: \"kubernetes.io/projected/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-kube-api-access-jgvjf\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.812024 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.811992 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:52.814519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:52.814493 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:53.062811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:53.062727 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:31:53.434242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:53.434168 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq"] Oct 30 22:31:53.440656 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:31:53.440627 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c240dd4_fd65_4f9d_89d6_593d39ae0ebc.slice/crio-ebb07d4fab3e22d0efdfaf068611b31bcf16a4a88459ce78d61409be1dafadfe WatchSource:0}: Error finding container ebb07d4fab3e22d0efdfaf068611b31bcf16a4a88459ce78d61409be1dafadfe: Status 404 returned error can't find the container with id ebb07d4fab3e22d0efdfaf068611b31bcf16a4a88459ce78d61409be1dafadfe Oct 30 22:31:53.442409 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:53.442390 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:31:53.904340 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:53.904303 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" event={"ID":"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc","Type":"ContainerStarted","Data":"ebb07d4fab3e22d0efdfaf068611b31bcf16a4a88459ce78d61409be1dafadfe"} Oct 30 22:31:54.914576 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:54.914531 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" event={"ID":"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc","Type":"ContainerStarted","Data":"51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760"} Oct 30 22:31:54.915817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:31:54.915792 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:32:01.931360 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:01.931325 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:32:02.044613 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:02.044545 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" podStartSLOduration=9.144181822 podStartE2EDuration="10.044523577s" totalImagesPullingTime="900.341755ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:55.029783389 +0000 UTC m=+3223.621068888" watchObservedRunningTime="2026-10-30 22:32:02.044523577 +0000 UTC m=+3230.635809067" Oct 30 22:32:15.282057 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.282027 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms"] Oct 30 22:32:15.286170 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.286150 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.335440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.335388 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-c29159-serving-cert\"" Oct 30 22:32:15.383321 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.383292 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.383452 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.383407 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0124e363-c519-4839-8e70-6a01675c1fa6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.383511 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.383473 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.383787 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.383768 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkt2\" (UniqueName: \"kubernetes.io/projected/0124e363-c519-4839-8e70-6a01675c1fa6-kube-api-access-lvkt2\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.384025 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.384005 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms"] Oct 30 22:32:15.484603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.484566 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkt2\" (UniqueName: \"kubernetes.io/projected/0124e363-c519-4839-8e70-6a01675c1fa6-kube-api-access-lvkt2\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.484795 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.484626 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.484795 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.484668 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0124e363-c519-4839-8e70-6a01675c1fa6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.484795 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.484691 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.484996 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:32:15.484856 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-serving-cert: secret "maas-api-e2e-shared-b-w0-c29159-serving-cert" not found Oct 30 22:32:15.484996 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:32:15.484920 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls podName:0124e363-c519-4839-8e70-6a01675c1fa6 nodeName:}" failed. No retries permitted until 2026-10-30 22:32:15.9849047 +0000 UTC m=+3244.576190178 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls") pod "maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" (UID: "0124e363-c519-4839-8e70-6a01675c1fa6") : secret "maas-api-e2e-shared-b-w0-c29159-serving-cert" not found Oct 30 22:32:15.485323 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.485302 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0124e363-c519-4839-8e70-6a01675c1fa6-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.487033 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.487013 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.555878 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.555798 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkt2\" (UniqueName: \"kubernetes.io/projected/0124e363-c519-4839-8e70-6a01675c1fa6-kube-api-access-lvkt2\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.990547 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.990522 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:15.992950 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:15.992931 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:16.199710 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:16.199675 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:16.572673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:16.572600 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms"] Oct 30 22:32:16.578397 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:32:16.578367 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0124e363_c519_4839_8e70_6a01675c1fa6.slice/crio-c72e7659dbc4b3ed774016f5dfb020a073c300337a32bf7fbdd7e2681ed369f1 WatchSource:0}: Error finding container c72e7659dbc4b3ed774016f5dfb020a073c300337a32bf7fbdd7e2681ed369f1: Status 404 returned error can't find the container with id c72e7659dbc4b3ed774016f5dfb020a073c300337a32bf7fbdd7e2681ed369f1 Oct 30 22:32:17.038088 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:17.038057 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" event={"ID":"0124e363-c519-4839-8e70-6a01675c1fa6","Type":"ContainerStarted","Data":"c72e7659dbc4b3ed774016f5dfb020a073c300337a32bf7fbdd7e2681ed369f1"} Oct 30 22:32:18.045636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:18.045599 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" event={"ID":"0124e363-c519-4839-8e70-6a01675c1fa6","Type":"ContainerStarted","Data":"afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75"} Oct 30 22:32:18.046088 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:18.045743 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:18.157332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:18.157278 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" podStartSLOduration=2.328162489 podStartE2EDuration="3.157265558s" totalImagesPullingTime="829.103069ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:18.156450844 +0000 UTC m=+3246.747736328" watchObservedRunningTime="2026-10-30 22:32:18.157265558 +0000 UTC m=+3246.748551044" Oct 30 22:32:19.804867 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:32:19.804825 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:32:24.057462 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:24.057383 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:32:49.805625 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:32:49.805579 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:32:52.727404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:52.727370 5139 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:32:52.727802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:52.727456 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:32:53.277020 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:53.276973 5139 generic.go:356] "Generic (PLEG): container finished" podID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerID="ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420" exitCode=0 Oct 30 22:32:53.277249 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:53.277229 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" event={"ID":"6c79e9de-5f9e-497f-8892-1a7341d92672","Type":"ContainerDied","Data":"ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420"} Oct 30 22:32:53.277820 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:53.277796 5139 scope.go:122] "RemoveContainer" containerID="ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420" Oct 30 22:32:53.287902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:53.287845 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:53.288038 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:53.287965 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:53.288038 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:53.288021 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:32:54.314438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:54.311393 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" event={"ID":"6c79e9de-5f9e-497f-8892-1a7341d92672","Type":"ContainerStarted","Data":"185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec"} Oct 30 22:32:54.996440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:54.986765 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:32:54.996440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:54.986898 5139 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:32:54.996440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:54.986946 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:32:55.986439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:55.985791 5139 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:32:55.986439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:55.985859 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:32:56.987822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:56.987787 5139 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:32:56.988464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:56.988392 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:32:57.988438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:57.987845 5139 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:32:57.988438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:57.987917 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:32:58.993440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:58.988846 5139 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" start-of-body= Oct 30 22:32:58.993440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:58.988923 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.118:15021/healthz/ready\": dial tcp 10.134.0.118:15021: connect: connection refused" Oct 30 22:32:59.990478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:59.990442 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:32:59.990914 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:59.990891 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:32:59.991604 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:32:59.991581 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:33:02.349649 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:02.348142 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": read tcp 10.134.0.2:48634->10.134.0.117:15021: read: connection reset by peer" start-of-body= Oct 30 22:33:02.349649 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:02.348214 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": read tcp 10.134.0.2:48634->10.134.0.117:15021: read: connection reset by peer" Oct 30 22:33:02.372089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:02.372059 5139 generic.go:356] "Generic (PLEG): container finished" podID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerID="7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d" exitCode=0 Oct 30 22:33:02.372229 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:02.372135 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" event={"ID":"8da5aadf-467a-4499-ae82-f7e05e2bde19","Type":"ContainerDied","Data":"7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d"} Oct 30 22:33:02.372628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:02.372609 5139 scope.go:122] "RemoveContainer" containerID="7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d" Oct 30 22:33:02.374219 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:02.374186 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:02.374274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:02.374258 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:02.374316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:02.374294 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:33:03.394855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:03.394777 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" event={"ID":"8da5aadf-467a-4499-ae82-f7e05e2bde19","Type":"ContainerStarted","Data":"42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554"} Oct 30 22:33:03.897839 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:03.897803 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:33:03.898032 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:03.897971 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:33:03.898032 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:03.898019 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:33:04.897668 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:04.897635 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:33:04.898171 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:04.897694 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:33:05.897956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:05.897921 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:33:05.898381 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:05.897982 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:33:06.897915 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:06.897876 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:33:06.898120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:06.897936 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:33:07.898036 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:07.898001 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:33:07.898211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:07.898064 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:33:08.901666 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:08.901637 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:33:08.902136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:08.902123 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:33:08.902868 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:33:08.902847 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:33:19.804876 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:33:19.804834 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:33:49.804859 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:33:49.804821 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:34:19.805593 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:34:19.805545 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:34:37.732913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:37.732881 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg"] Oct 30 22:34:37.733349 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:37.733128 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" containerID="cri-o://068a5630d1203cbb62548ed309c33a6b9299a39299c4e6c6c1192798806ad214" gracePeriod=30 Oct 30 22:34:37.779617 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:37.779583 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4"] Oct 30 22:34:37.779845 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:37.779822 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" containerID="cri-o://99e6188aa46dc7834b6a1b1fe7f4426254e5c84a9b80eaf91a45b180dd6de7cc" gracePeriod=30 Oct 30 22:34:37.781842 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:37.781812 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll"] Oct 30 22:34:37.782006 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:37.781988 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" containerID="cri-o://623117ebce44bddf4661ac3cd87bb284ccf31a0b2e804c1e077d24afee4b0e9e" gracePeriod=30 Oct 30 22:34:42.512741 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:42.512700 5139 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:34:42.513224 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:42.512780 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:34:43.043775 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.043511 5139 generic.go:356] "Generic (PLEG): container finished" podID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerID="99e6188aa46dc7834b6a1b1fe7f4426254e5c84a9b80eaf91a45b180dd6de7cc" exitCode=0 Oct 30 22:34:43.043775 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.043696 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" event={"ID":"8fdbd86c-47fe-48ba-8e68-64706316e2c1","Type":"ContainerDied","Data":"99e6188aa46dc7834b6a1b1fe7f4426254e5c84a9b80eaf91a45b180dd6de7cc"} Oct 30 22:34:43.045408 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.045383 5139 generic.go:356] "Generic (PLEG): container finished" podID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerID="068a5630d1203cbb62548ed309c33a6b9299a39299c4e6c6c1192798806ad214" exitCode=0 Oct 30 22:34:43.045552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.045465 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" event={"ID":"ada185ee-3ddd-43ac-bdfa-b29e211193be","Type":"ContainerDied","Data":"068a5630d1203cbb62548ed309c33a6b9299a39299c4e6c6c1192798806ad214"} Oct 30 22:34:43.045685 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.045553 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" event={"ID":"ada185ee-3ddd-43ac-bdfa-b29e211193be","Type":"ContainerDied","Data":"954340746a1cc42d86ec1ae70eb98d5fb96abff87c30b7f5d756b0fb75214824"} Oct 30 22:34:43.045685 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.045578 5139 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="954340746a1cc42d86ec1ae70eb98d5fb96abff87c30b7f5d756b0fb75214824" Oct 30 22:34:43.048501 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.048378 5139 generic.go:356] "Generic (PLEG): container finished" podID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerID="623117ebce44bddf4661ac3cd87bb284ccf31a0b2e804c1e077d24afee4b0e9e" exitCode=0 Oct 30 22:34:43.048501 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.048435 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" event={"ID":"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f","Type":"ContainerDied","Data":"623117ebce44bddf4661ac3cd87bb284ccf31a0b2e804c1e077d24afee4b0e9e"} Oct 30 22:34:43.059275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.059257 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:34:43.091844 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.091822 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:34:43.099162 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.099142 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:34:43.151225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151204 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-certs\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151331 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151263 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-podinfo\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151331 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151304 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-data\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151475 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151366 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-envoy\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151475 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151439 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-kube-api-access-d88t6\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-kube-api-access-d88t6\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151588 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151474 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-certs" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.151588 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151512 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-credential-socket\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151736 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151588 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ada185ee-3ddd-43ac-bdfa-b29e211193be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada185ee-3ddd-43ac-bdfa-b29e211193be-istiod-ca-cert\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151736 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151631 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-socket\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151736 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151650 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-data" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.151736 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151667 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-token\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-token\") pod \"ada185ee-3ddd-43ac-bdfa-b29e211193be\" (UID: \"ada185ee-3ddd-43ac-bdfa-b29e211193be\") " Oct 30 22:34:43.151953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151813 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-credential-socket" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.151953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151870 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-socket" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.151953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.151904 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ada185ee-3ddd-43ac-bdfa-b29e211193be-istiod-ca-cert" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:43.156298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.152845 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.156298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.152891 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ada185ee-3ddd-43ac-bdfa-b29e211193be-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.156298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.152908 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.156298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.152924 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.156298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.152944 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.162094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.162064 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-token" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:43.162094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.162081 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-envoy" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.162258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.162104 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-kube-api-access-d88t6" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "kube-api-access-d88t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:43.162258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.162099 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-podinfo" pod "ada185ee-3ddd-43ac-bdfa-b29e211193be" (UID: "ada185ee-3ddd-43ac-bdfa-b29e211193be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:43.254187 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254156 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-credential-socket\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.254345 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254191 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-envoy\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.254345 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254226 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-podinfo\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.254345 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254305 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-data\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.254592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254351 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-certs\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.254592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254399 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istiod-ca-cert\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.254592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254451 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-token\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-token\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.254592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254469 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-credential-socket" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.254592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254490 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-data\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.254592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254551 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-token\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-token\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254599 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-podinfo\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254629 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-credential-socket\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254631 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-data" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254674 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-certs" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254707 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istiod-ca-cert\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254764 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-socket\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254799 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-certs\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254833 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-kube-api-access-hzrk8\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-kube-api-access-hzrk8\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254871 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-socket\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.254918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254909 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-kube-api-access-4md2x\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-kube-api-access-4md2x\") pod \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\" (UID: \"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f\") " Oct 30 22:34:43.255472 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254938 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-envoy\") pod \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\" (UID: \"8fdbd86c-47fe-48ba-8e68-64706316e2c1\") " Oct 30 22:34:43.255580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255551 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.255580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255574 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.255697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255591 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.255697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255606 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.255697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255620 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.255697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255634 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ada185ee-3ddd-43ac-bdfa-b29e211193be-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.255697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255648 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d88t6\" (UniqueName: \"kubernetes.io/projected/ada185ee-3ddd-43ac-bdfa-b29e211193be-kube-api-access-d88t6\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.256184 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254795 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-data" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.256271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.254818 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istiod-ca-cert" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:43.256271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255007 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-credential-socket" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.256271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255083 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-socket" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.256271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.255361 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istiod-ca-cert" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:34:43.256271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.256150 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-certs" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.256585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.256406 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-socket" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.258190 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.258159 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-token" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:43.258303 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.258191 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-kube-api-access-hzrk8" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "kube-api-access-hzrk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:43.258398 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.258367 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-podinfo" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:43.258526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.258371 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-envoy" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.258789 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.258766 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-podinfo" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:34:43.258789 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.258775 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-envoy" pod "8fdbd86c-47fe-48ba-8e68-64706316e2c1" (UID: "8fdbd86c-47fe-48ba-8e68-64706316e2c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:34:43.258964 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.258947 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-kube-api-access-4md2x" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "kube-api-access-4md2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:43.259027 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.259007 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-token" pod "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" (UID: "5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357194 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357234 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357245 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357254 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357262 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357271 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357279 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzrk8\" (UniqueName: \"kubernetes.io/projected/8fdbd86c-47fe-48ba-8e68-64706316e2c1-kube-api-access-hzrk8\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357288 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357297 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4md2x\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-kube-api-access-4md2x\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357306 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357968 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357314 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357968 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357323 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357968 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357332 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357968 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357339 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:43.357968 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:43.357350 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fdbd86c-47fe-48ba-8e68-64706316e2c1-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:34:44.056645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.056567 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" Oct 30 22:34:44.056645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.056579 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll" event={"ID":"5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f","Type":"ContainerDied","Data":"b4d5a4760f5d16d3c287e281b421a0f3943f3e9efc05aaf5edb03cdf2bb6ceaa"} Oct 30 22:34:44.056645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.056642 5139 scope.go:122] "RemoveContainer" containerID="623117ebce44bddf4661ac3cd87bb284ccf31a0b2e804c1e077d24afee4b0e9e" Oct 30 22:34:44.058094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.058073 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg" Oct 30 22:34:44.058177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.058097 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" event={"ID":"8fdbd86c-47fe-48ba-8e68-64706316e2c1","Type":"ContainerDied","Data":"ec72fc042c98d46ea25aff7123646e8c80fb3ca31104f635af4021783b64d0aa"} Oct 30 22:34:44.058177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.058073 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4" Oct 30 22:34:44.067028 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.067009 5139 scope.go:122] "RemoveContainer" containerID="99e6188aa46dc7834b6a1b1fe7f4426254e5c84a9b80eaf91a45b180dd6de7cc" Oct 30 22:34:44.212253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.212222 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg"] Oct 30 22:34:44.258725 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.258696 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-vhcmg"] Oct 30 22:34:44.321753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.321699 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll"] Oct 30 22:34:44.328761 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.328738 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-s4kll"] Oct 30 22:34:44.431067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.431039 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4"] Oct 30 22:34:44.435308 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:44.435285 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-m62w4"] Oct 30 22:34:45.543859 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:45.543825 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" path="/var/lib/kubelet/pods/5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f/volumes" Oct 30 22:34:45.544470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:45.544454 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" path="/var/lib/kubelet/pods/8fdbd86c-47fe-48ba-8e68-64706316e2c1/volumes" Oct 30 22:34:45.545032 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:45.545018 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" path="/var/lib/kubelet/pods/ada185ee-3ddd-43ac-bdfa-b29e211193be/volumes" Oct 30 22:34:49.804620 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:34:49.804577 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:34:59.022224 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:59.022182 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz"] Oct 30 22:34:59.025720 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:34:59.025639 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" containerID="cri-o://38d77984df9779db5c423b49edbfbf16183526a47f5d365342bfe6e6e59143bd" gracePeriod=30 Oct 30 22:35:04.178484 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.178439 5139 generic.go:356] "Generic (PLEG): container finished" podID="495e4ea7-3763-412a-a131-40f30023e5da" containerID="38d77984df9779db5c423b49edbfbf16183526a47f5d365342bfe6e6e59143bd" exitCode=0 Oct 30 22:35:04.179054 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.178509 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" event={"ID":"495e4ea7-3763-412a-a131-40f30023e5da","Type":"ContainerDied","Data":"38d77984df9779db5c423b49edbfbf16183526a47f5d365342bfe6e6e59143bd"} Oct 30 22:35:04.293873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.293815 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:35:04.394755 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.394718 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/495e4ea7-3763-412a-a131-40f30023e5da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495e4ea7-3763-412a-a131-40f30023e5da-istiod-ca-cert\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.394964 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.394775 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-kube-api-access-46bb5\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-kube-api-access-46bb5\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.394964 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.394824 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-certs\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.394964 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.394841 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-credential-socket\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.395161 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395046 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-data\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.395161 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395128 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/495e4ea7-3763-412a-a131-40f30023e5da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495e4ea7-3763-412a-a131-40f30023e5da-istio-podinfo\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.395269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395184 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-certs" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:04.395269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395204 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-credential-socket" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:04.395269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395194 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-envoy\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.395450 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395294 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-istio-token\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-istio-token\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.395450 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395225 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/495e4ea7-3763-412a-a131-40f30023e5da-istiod-ca-cert" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:04.395450 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395313 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-data" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:04.395450 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395365 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-socket\") pod \"495e4ea7-3763-412a-a131-40f30023e5da\" (UID: \"495e4ea7-3763-412a-a131-40f30023e5da\") " Oct 30 22:35:04.395717 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.395665 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-socket" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:04.396043 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.396018 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/495e4ea7-3763-412a-a131-40f30023e5da-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.396143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.396048 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.396143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.396065 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.396143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.396080 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.396143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.396095 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.397629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.397599 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/495e4ea7-3763-412a-a131-40f30023e5da-istio-podinfo" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:04.397756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.397675 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-envoy" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:04.397756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.397704 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-kube-api-access-46bb5" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "kube-api-access-46bb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:04.399532 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.399510 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-istio-token" pod "495e4ea7-3763-412a-a131-40f30023e5da" (UID: "495e4ea7-3763-412a-a131-40f30023e5da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:04.496848 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.496808 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/495e4ea7-3763-412a-a131-40f30023e5da-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.496848 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.496837 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/495e4ea7-3763-412a-a131-40f30023e5da-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.496848 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.496846 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:04.496848 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:04.496856 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46bb5\" (UniqueName: \"kubernetes.io/projected/495e4ea7-3763-412a-a131-40f30023e5da-kube-api-access-46bb5\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:05.185261 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:05.185230 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" Oct 30 22:35:05.185765 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:05.185227 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz" event={"ID":"495e4ea7-3763-412a-a131-40f30023e5da","Type":"ContainerDied","Data":"636f7dd098726d0e535fe2d66fcb4aa20a9b81c0463b66c2c457a8de141d1297"} Oct 30 22:35:05.185765 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:05.185382 5139 scope.go:122] "RemoveContainer" containerID="38d77984df9779db5c423b49edbfbf16183526a47f5d365342bfe6e6e59143bd" Oct 30 22:35:05.387912 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:05.387878 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz"] Oct 30 22:35:05.434433 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:05.434382 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-69bpz"] Oct 30 22:35:05.543327 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:05.543296 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="495e4ea7-3763-412a-a131-40f30023e5da" path="/var/lib/kubelet/pods/495e4ea7-3763-412a-a131-40f30023e5da/volumes" Oct 30 22:35:12.354775 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:12.354744 5139 scope.go:122] "RemoveContainer" containerID="068a5630d1203cbb62548ed309c33a6b9299a39299c4e6c6c1192798806ad214" Oct 30 22:35:13.253074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:13.253041 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8"] Oct 30 22:35:13.253315 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:13.253290 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" containerID="cri-o://acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb" gracePeriod=30 Oct 30 22:35:17.943490 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:17.943449 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:35:17.943879 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:17.943519 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:35:18.503603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.503575 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:35:18.664989 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.664906 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-credential-socket\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665152 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.664992 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istiod-ca-cert\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665152 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665082 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-data\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665237 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665156 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-podinfo\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665237 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665187 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-certs\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665237 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665212 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-kube-api-access-gtxg4\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-kube-api-access-gtxg4\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665243 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-token\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-token\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665246 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-credential-socket" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:18.665395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665300 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-socket\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665395 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665344 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-envoy\") pod \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\" (UID: \"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5\") " Oct 30 22:35:18.665626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665531 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istiod-ca-cert" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:35:18.665626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665544 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-certs" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:18.665778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665747 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-data" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:18.665778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.665758 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-socket" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:18.666332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.666079 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:18.666332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.666104 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:18.666332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.666122 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:18.666332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.666137 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:18.666332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.666153 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:18.667523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.667496 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-podinfo" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:35:18.667908 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.667885 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-kube-api-access-gtxg4" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "kube-api-access-gtxg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:18.667987 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.667963 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-envoy" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:35:18.668107 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.668086 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-token" pod "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" (UID: "107566ff-9d1c-4ae2-be87-6adeaf7cbaa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:35:18.767328 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.767291 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtxg4\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-kube-api-access-gtxg4\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:18.767328 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.767322 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:18.767555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.767339 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:18.767555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:18.767352 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:35:19.264511 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.264474 5139 generic.go:356] "Generic (PLEG): container finished" podID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerID="acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb" exitCode=0 Oct 30 22:35:19.265006 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.264529 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" Oct 30 22:35:19.265006 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.264555 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" event={"ID":"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5","Type":"ContainerDied","Data":"acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb"} Oct 30 22:35:19.265006 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.264614 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8" event={"ID":"107566ff-9d1c-4ae2-be87-6adeaf7cbaa5","Type":"ContainerDied","Data":"f43643c0d1ab393805532b84f352f41d8e49eb68f69ea77d620a43733cda74ab"} Oct 30 22:35:19.265006 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.264638 5139 scope.go:122] "RemoveContainer" containerID="acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb" Oct 30 22:35:19.274650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.274633 5139 scope.go:122] "RemoveContainer" containerID="acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb" Oct 30 22:35:19.274897 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:35:19.274878 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb\": container with ID starting with acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb not found: ID does not exist" containerID="acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb" Oct 30 22:35:19.274954 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.274909 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb"} err="failed to get container status \"acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb\": rpc error: code = NotFound desc = could not find container \"acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb\": container with ID starting with acbc4eafad1e236010f82d4821e58629b5cbcd5041320fd30801aa54567f77cb not found: ID does not exist" Oct 30 22:35:19.421766 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.421732 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8"] Oct 30 22:35:19.468162 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.468130 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-8j4r8"] Oct 30 22:35:19.543600 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:19.543518 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" path="/var/lib/kubelet/pods/107566ff-9d1c-4ae2-be87-6adeaf7cbaa5/volumes" Oct 30 22:35:19.804564 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:35:19.804481 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:35:49.808271 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:35:49.808227 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:35:50.181975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.181892 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6"] Oct 30 22:35:50.182428 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182404 5139 cpu_manager.go:414] "containerMap: removing container" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" Oct 30 22:35:50.182483 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182432 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" Oct 30 22:35:50.182483 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182445 5139 cpu_manager.go:414] "containerMap: removing container" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" Oct 30 22:35:50.182483 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182450 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" Oct 30 22:35:50.182483 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182467 5139 cpu_manager.go:414] "containerMap: removing container" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" Oct 30 22:35:50.182483 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182472 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182488 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182493 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182513 5139 cpu_manager.go:414] "containerMap: removing container" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182519 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182601 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fdbd86c-47fe-48ba-8e68-64706316e2c1" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182610 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="495e4ea7-3763-412a-a131-40f30023e5da" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182618 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f1e1e49-35b4-4af9-ac05-3ac2c2ac243f" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182626 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ada185ee-3ddd-43ac-bdfa-b29e211193be" containerName="istio-proxy" Oct 30 22:35:50.182634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.182634 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="107566ff-9d1c-4ae2-be87-6adeaf7cbaa5" containerName="istio-proxy" Oct 30 22:35:50.185384 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.185364 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.234053 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.234015 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tls-certs\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.234196 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.234147 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj5pp\" (UniqueName: \"kubernetes.io/projected/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-kube-api-access-gj5pp\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.234251 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.234225 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-model-cache\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.234319 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.234303 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-dshm\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.234401 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.234385 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tmp-dir\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.234596 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.234577 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-home\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.286638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.286607 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae7a1f\"/\"openshift-service-ca.crt\"" Oct 30 22:35:50.286638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.286633 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae7a1f\"/\"kube-root-ca.crt\"" Oct 30 22:35:50.286843 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.286648 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae7a1f\"/\"auth-test-model-ae7a1f-kserve-self-signed-certs\"" Oct 30 22:35:50.286843 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.286614 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-ae7a1f\"/\"default-dockercfg-rtzg2\"" Oct 30 22:35:50.290289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.290268 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6"] Oct 30 22:35:50.336126 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.336090 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj5pp\" (UniqueName: \"kubernetes.io/projected/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-kube-api-access-gj5pp\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.336389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.336132 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-model-cache\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.336389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.336160 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-dshm\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.336389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.336226 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tmp-dir\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.336681 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.336481 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-home\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.336681 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.336569 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tls-certs\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.338274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.338248 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tmp-dir\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.339188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.338717 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-model-cache\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.340534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.340246 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-home\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.342591 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.342566 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tls-certs\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.344900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.344013 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-dshm\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.400060 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.400023 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj5pp\" (UniqueName: \"kubernetes.io/projected/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-kube-api-access-gj5pp\") pod \"auth-test-model-ae7a1f-kserve-78dc648477-xz2t6\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.496883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.496858 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:50.875461 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:50.875361 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6"] Oct 30 22:35:50.881848 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:35:50.881820 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf4dc40bf_9912_49ff_a7eb_4ef8e6cddaa4.slice/crio-9c09451cf75d82580e3a3e8539b1a370325ed11d74b8ba1699988a6e8d29b139 WatchSource:0}: Error finding container 9c09451cf75d82580e3a3e8539b1a370325ed11d74b8ba1699988a6e8d29b139: Status 404 returned error can't find the container with id 9c09451cf75d82580e3a3e8539b1a370325ed11d74b8ba1699988a6e8d29b139 Oct 30 22:35:51.443125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:51.443091 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" event={"ID":"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4","Type":"ContainerStarted","Data":"910b73b90cfa452e5345016e3b856528c8651b9ef31ae3602605dd5dc1923d23"} Oct 30 22:35:51.443304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:51.443135 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" event={"ID":"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4","Type":"ContainerStarted","Data":"9c09451cf75d82580e3a3e8539b1a370325ed11d74b8ba1699988a6e8d29b139"} Oct 30 22:35:51.497761 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:51.497734 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:35:51.555307 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:35:51.555261 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" podStartSLOduration=1.555248076 podStartE2EDuration="1.555248076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:51.553681594 +0000 UTC m=+3460.144967081" watchObservedRunningTime="2026-10-30 22:35:51.555248076 +0000 UTC m=+3460.146533563" Oct 30 22:36:00.497275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:00.497238 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:36:00.510477 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:00.510449 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:36:01.508713 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:01.508682 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:36:06.277787 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:06.277752 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l"] Oct 30 22:36:06.278298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:06.277969 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" containerID="cri-o://185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec" gracePeriod=30 Oct 30 22:36:07.457215 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.457175 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq"] Oct 30 22:36:07.461067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.461045 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.514960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.514925 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-970415-openshift-default-dockercfg-cdxg6\"" Oct 30 22:36:07.548122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548084 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b42527-6fc7-4baf-8bc8-b0a494e11461-istiod-ca-cert\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.548122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548138 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-credential-socket\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.548370 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548181 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-podinfo\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.548370 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548210 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-socket\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.548370 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548270 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-certs\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.548370 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548348 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-envoy\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.548564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548388 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-token\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.548564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548450 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-data\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.548564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.548500 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl7mk\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-kube-api-access-wl7mk\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.564645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.564610 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq"] Oct 30 22:36:07.649470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649426 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-podinfo\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.649662 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649481 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-socket\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.649662 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649559 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-certs\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.649662 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649659 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-envoy\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.649840 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649704 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-token\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.649840 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649775 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-data\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.649947 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649863 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl7mk\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-kube-api-access-wl7mk\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.650003 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649985 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-socket\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.650078 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.649996 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-certs\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.650232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.650214 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-data\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.650325 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.650259 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b42527-6fc7-4baf-8bc8-b0a494e11461-istiod-ca-cert\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.650325 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.650308 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-credential-socket\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.650638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.650616 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-credential-socket\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.650942 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.650923 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b42527-6fc7-4baf-8bc8-b0a494e11461-istiod-ca-cert\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.652932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.652909 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-envoy\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.655926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.655899 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-podinfo\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.709796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.709718 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-token\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.709951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.709854 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl7mk\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-kube-api-access-wl7mk\") pod \"e2e-noten-970415-openshift-default-658c8594d5-rddlq\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:07.774091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:07.774060 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:08.211731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:08.211691 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq"] Oct 30 22:36:08.220241 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:36:08.220212 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82b42527_6fc7_4baf_8bc8_b0a494e11461.slice/crio-d952a37da75d51a3a11888896754f77abc814543ca86b72a6ea42692a384d459 WatchSource:0}: Error finding container d952a37da75d51a3a11888896754f77abc814543ca86b72a6ea42692a384d459: Status 404 returned error can't find the container with id d952a37da75d51a3a11888896754f77abc814543ca86b72a6ea42692a384d459 Oct 30 22:36:08.223207 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:08.223168 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:36:08.223297 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:08.223272 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:36:08.223341 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:08.223305 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:36:08.554758 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:08.554726 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" event={"ID":"82b42527-6fc7-4baf-8bc8-b0a494e11461","Type":"ContainerStarted","Data":"622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab"} Oct 30 22:36:08.555156 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:08.554773 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" event={"ID":"82b42527-6fc7-4baf-8bc8-b0a494e11461","Type":"ContainerStarted","Data":"d952a37da75d51a3a11888896754f77abc814543ca86b72a6ea42692a384d459"} Oct 30 22:36:08.669989 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:08.669941 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" podStartSLOduration=1.6699271580000001 podStartE2EDuration="1.669927158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:36:08.667704575 +0000 UTC m=+3477.258990359" watchObservedRunningTime="2026-10-30 22:36:08.669927158 +0000 UTC m=+3477.261212712" Oct 30 22:36:08.774318 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:08.774283 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:09.781535 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:09.781502 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:10.565012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:10.564977 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:10.565967 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:10.565950 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:36:11.539722 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.539694 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:36:11.571304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.571216 5139 generic.go:356] "Generic (PLEG): container finished" podID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerID="185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec" exitCode=0 Oct 30 22:36:11.571304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.571282 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" Oct 30 22:36:11.571513 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.571306 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" event={"ID":"6c79e9de-5f9e-497f-8892-1a7341d92672","Type":"ContainerDied","Data":"185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec"} Oct 30 22:36:11.571513 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.571358 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l" event={"ID":"6c79e9de-5f9e-497f-8892-1a7341d92672","Type":"ContainerDied","Data":"3c3a6eb850f887a7b1bbca947f38ca1fed4fb4a5664e71720fb554f647c06a31"} Oct 30 22:36:11.571513 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.571381 5139 scope.go:122] "RemoveContainer" containerID="185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec" Oct 30 22:36:11.581251 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.581227 5139 scope.go:122] "RemoveContainer" containerID="ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420" Oct 30 22:36:11.590805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.590787 5139 scope.go:122] "RemoveContainer" containerID="185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec" Oct 30 22:36:11.591074 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:36:11.591055 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec\": container with ID starting with 185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec not found: ID does not exist" containerID="185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec" Oct 30 22:36:11.591161 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.591083 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec"} err="failed to get container status \"185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec\": rpc error: code = NotFound desc = could not find container \"185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec\": container with ID starting with 185f11bc7467a79e3ed1b3aef7175e141714abe60155876aec969fa868c725ec not found: ID does not exist" Oct 30 22:36:11.591161 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.591103 5139 scope.go:122] "RemoveContainer" containerID="ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420" Oct 30 22:36:11.591385 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:36:11.591367 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420\": container with ID starting with ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420 not found: ID does not exist" containerID="ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420" Oct 30 22:36:11.591499 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.591388 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420"} err="failed to get container status \"ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420\": rpc error: code = NotFound desc = could not find container \"ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420\": container with ID starting with ecb75de7525da9daa570de6ddfa1eebfe646853f45761bd88a3e30a3289d3420 not found: ID does not exist" Oct 30 22:36:11.696482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696445 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-kube-api-access-kk5p6\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-kube-api-access-kk5p6\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.696679 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696516 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-token\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-token\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.696679 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696544 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-data\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.696679 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696615 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-certs\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.696679 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696649 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-envoy\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.696920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696719 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c79e9de-5f9e-497f-8892-1a7341d92672-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79e9de-5f9e-497f-8892-1a7341d92672-istiod-ca-cert\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.696920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696750 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-credential-socket\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.696920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696793 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-podinfo\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.696920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696859 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-socket\") pod \"6c79e9de-5f9e-497f-8892-1a7341d92672\" (UID: \"6c79e9de-5f9e-497f-8892-1a7341d92672\") " Oct 30 22:36:11.697138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696944 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-data" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.697138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.696955 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-certs" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.697240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.697149 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c79e9de-5f9e-497f-8892-1a7341d92672-istiod-ca-cert" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:11.697240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.697168 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-credential-socket" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.697394 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.697375 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-socket" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.698086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.697883 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c79e9de-5f9e-497f-8892-1a7341d92672-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.698086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.698015 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.698086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.698040 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.698086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.698055 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.698086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.698070 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.699161 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.699129 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-podinfo" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:11.699161 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.699146 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-envoy" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:11.699330 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.699215 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-token" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:11.699501 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.699480 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-kube-api-access-kk5p6" pod "6c79e9de-5f9e-497f-8892-1a7341d92672" (UID: "6c79e9de-5f9e-497f-8892-1a7341d92672"). InnerVolumeSpecName "kube-api-access-kk5p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:11.799031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.799000 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.799031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.799031 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kk5p6\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-kube-api-access-kk5p6\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.799031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.799045 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.799271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.799059 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c79e9de-5f9e-497f-8892-1a7341d92672-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:11.987344 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.987309 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l"] Oct 30 22:36:11.992266 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:11.992240 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-dpv7l"] Oct 30 22:36:13.542809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:13.542773 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" path="/var/lib/kubelet/pods/6c79e9de-5f9e-497f-8892-1a7341d92672/volumes" Oct 30 22:36:19.804341 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:36:19.804303 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:36:43.931777 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:43.931744 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8"] Oct 30 22:36:43.932264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:43.931945 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" containerID="cri-o://d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d" gracePeriod=30 Oct 30 22:36:49.182163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.182137 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:36:49.333967 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.333937 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-envoy\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.333980 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0ca472a-aeec-4144-a7bf-516da8567642-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca472a-aeec-4144-a7bf-516da8567642-istiod-ca-cert\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334002 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-socket\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334040 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-certs\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334119 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0ca472a-aeec-4144-a7bf-516da8567642-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca472a-aeec-4144-a7bf-516da8567642-istio-podinfo\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334142 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-credential-socket\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334189 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-kube-api-access-snv5f\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-kube-api-access-snv5f\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334226 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-data\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334261 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-istio-token\") pod \"a0ca472a-aeec-4144-a7bf-516da8567642\" (UID: \"a0ca472a-aeec-4144-a7bf-516da8567642\") " Oct 30 22:36:49.334690 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334443 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-certs" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:49.334690 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334455 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-socket" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:49.334690 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334486 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-credential-socket" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:49.334690 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334516 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0ca472a-aeec-4144-a7bf-516da8567642-istiod-ca-cert" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:49.334903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334720 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-data" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:49.334903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334872 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0ca472a-aeec-4144-a7bf-516da8567642-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.334903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334893 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.335026 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334907 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.335026 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334921 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.335026 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.334938 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.336349 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.336327 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0ca472a-aeec-4144-a7bf-516da8567642-istio-podinfo" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:36:49.336472 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.336357 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-istio-token" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:49.336567 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.336551 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-envoy" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:49.336686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.336670 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-kube-api-access-snv5f" pod "a0ca472a-aeec-4144-a7bf-516da8567642" (UID: "a0ca472a-aeec-4144-a7bf-516da8567642"). InnerVolumeSpecName "kube-api-access-snv5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:49.436131 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.436103 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0ca472a-aeec-4144-a7bf-516da8567642-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.436131 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.436129 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0ca472a-aeec-4144-a7bf-516da8567642-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.436327 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.436141 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snv5f\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-kube-api-access-snv5f\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.436327 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.436152 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0ca472a-aeec-4144-a7bf-516da8567642-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:49.776746 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.776710 5139 generic.go:356] "Generic (PLEG): container finished" podID="a0ca472a-aeec-4144-a7bf-516da8567642" containerID="d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d" exitCode=0 Oct 30 22:36:49.776924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.776777 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" Oct 30 22:36:49.776924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.776816 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" event={"ID":"a0ca472a-aeec-4144-a7bf-516da8567642","Type":"ContainerDied","Data":"d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d"} Oct 30 22:36:49.776924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.776879 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8" event={"ID":"a0ca472a-aeec-4144-a7bf-516da8567642","Type":"ContainerDied","Data":"d05fe9c29e2081d6472d0dba089b96366dcf8c025b94c3d34335051dfec6bdd4"} Oct 30 22:36:49.776924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.776910 5139 scope.go:122] "RemoveContainer" containerID="d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d" Oct 30 22:36:49.788114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.788093 5139 scope.go:122] "RemoveContainer" containerID="d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d" Oct 30 22:36:49.788512 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:36:49.788393 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d\": container with ID starting with d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d not found: ID does not exist" containerID="d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d" Oct 30 22:36:49.788512 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.788436 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d"} err="failed to get container status \"d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d\": rpc error: code = NotFound desc = could not find container \"d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d\": container with ID starting with d8945b43aff816b23d26802d027c904ffd5685abe6b56a8b1f62297e22658b7d not found: ID does not exist" Oct 30 22:36:49.806728 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:36:49.806692 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:36:49.931892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.931858 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8"] Oct 30 22:36:49.978389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:49.978355 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-ck8v8"] Oct 30 22:36:51.543842 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:51.543764 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" path="/var/lib/kubelet/pods/a0ca472a-aeec-4144-a7bf-516da8567642/volumes" Oct 30 22:36:58.715342 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:58.715303 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr"] Oct 30 22:36:58.715773 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:58.715503 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" podUID="d9bf96bc-6f37-4a91-aa39-4a988b797bd8" containerName="discovery" containerID="cri-o://8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987" gracePeriod=30 Oct 30 22:36:58.980842 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:58.980758 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:36:59.047296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.047265 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-dns-cert\") pod \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " Oct 30 22:36:59.047493 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.047320 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-kube-api-access-p8gqp\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-kube-api-access-p8gqp\") pod \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " Oct 30 22:36:59.047493 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.047376 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-local-certs\") pod \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " Oct 30 22:36:59.047493 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.047395 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-kubeconfig\") pod \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " Oct 30 22:36:59.047493 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.047470 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-token\") pod \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " Oct 30 22:36:59.047698 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.047535 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-cacerts\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-cacerts\") pod \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " Oct 30 22:36:59.047698 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.047581 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-ca-configmap\") pod \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\" (UID: \"d9bf96bc-6f37-4a91-aa39-4a988b797bd8\") " Oct 30 22:36:59.048391 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.048355 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-ca-configmap" pod "d9bf96bc-6f37-4a91-aa39-4a988b797bd8" (UID: "d9bf96bc-6f37-4a91-aa39-4a988b797bd8"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:36:59.049817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.049786 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-token" pod "d9bf96bc-6f37-4a91-aa39-4a988b797bd8" (UID: "d9bf96bc-6f37-4a91-aa39-4a988b797bd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:59.049918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.049861 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-cacerts" pod "d9bf96bc-6f37-4a91-aa39-4a988b797bd8" (UID: "d9bf96bc-6f37-4a91-aa39-4a988b797bd8"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:59.049975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.049916 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-kube-api-access-p8gqp" pod "d9bf96bc-6f37-4a91-aa39-4a988b797bd8" (UID: "d9bf96bc-6f37-4a91-aa39-4a988b797bd8"). InnerVolumeSpecName "kube-api-access-p8gqp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:36:59.049975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.049933 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-dns-cert" pod "d9bf96bc-6f37-4a91-aa39-4a988b797bd8" (UID: "d9bf96bc-6f37-4a91-aa39-4a988b797bd8"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:59.050086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.050027 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-kubeconfig" pod "d9bf96bc-6f37-4a91-aa39-4a988b797bd8" (UID: "d9bf96bc-6f37-4a91-aa39-4a988b797bd8"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:36:59.050086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.050036 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-local-certs" pod "d9bf96bc-6f37-4a91-aa39-4a988b797bd8" (UID: "d9bf96bc-6f37-4a91-aa39-4a988b797bd8"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:36:59.149357 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.149326 5139 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-dns-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:59.149357 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.149354 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8gqp\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-kube-api-access-p8gqp\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:59.149357 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.149365 5139 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-local-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:59.149609 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.149373 5139 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-kubeconfig\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:59.149609 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.149382 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:59.149609 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.149390 5139 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-cacerts\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:59.149609 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.149398 5139 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d9bf96bc-6f37-4a91-aa39-4a988b797bd8-istio-csr-ca-configmap\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:36:59.832597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.832500 5139 generic.go:356] "Generic (PLEG): container finished" podID="d9bf96bc-6f37-4a91-aa39-4a988b797bd8" containerID="8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987" exitCode=0 Oct 30 22:36:59.832597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.832563 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" Oct 30 22:36:59.833144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.832562 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" event={"ID":"d9bf96bc-6f37-4a91-aa39-4a988b797bd8","Type":"ContainerDied","Data":"8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987"} Oct 30 22:36:59.833144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.832689 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr" event={"ID":"d9bf96bc-6f37-4a91-aa39-4a988b797bd8","Type":"ContainerDied","Data":"d59b075708cf95b22aed27db9947fb24a25b1e891bbf3181d3996b290606161d"} Oct 30 22:36:59.833144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.832708 5139 scope.go:122] "RemoveContainer" containerID="8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987" Oct 30 22:36:59.841936 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.841910 5139 scope.go:122] "RemoveContainer" containerID="8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987" Oct 30 22:36:59.842196 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:36:59.842177 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987\": container with ID starting with 8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987 not found: ID does not exist" containerID="8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987" Oct 30 22:36:59.842249 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:36:59.842204 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987"} err="failed to get container status \"8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987\": rpc error: code = NotFound desc = could not find container \"8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987\": container with ID starting with 8186ecbdd23f1678a05e1ea59aff66ab803bdcd9f63f0b828374471f006ce987 not found: ID does not exist" Oct 30 22:37:00.034953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:00.034917 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr"] Oct 30 22:37:00.081016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:00.080977 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-lf2mr"] Oct 30 22:37:01.542906 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:01.542873 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bf96bc-6f37-4a91-aa39-4a988b797bd8" path="/var/lib/kubelet/pods/d9bf96bc-6f37-4a91-aa39-4a988b797bd8/volumes" Oct 30 22:37:04.943021 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.942982 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk"] Oct 30 22:37:04.943752 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943729 5139 cpu_manager.go:414] "containerMap: removing container" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" Oct 30 22:37:04.943850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943753 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" Oct 30 22:37:04.943850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943778 5139 cpu_manager.go:414] "containerMap: removing container" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" Oct 30 22:37:04.943850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943785 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" Oct 30 22:37:04.943850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943808 5139 cpu_manager.go:414] "containerMap: removing container" podUID="d9bf96bc-6f37-4a91-aa39-4a988b797bd8" containerName="discovery" Oct 30 22:37:04.943850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943816 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bf96bc-6f37-4a91-aa39-4a988b797bd8" containerName="discovery" Oct 30 22:37:04.944100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943969 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" Oct 30 22:37:04.944100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943983 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0ca472a-aeec-4144-a7bf-516da8567642" containerName="istio-proxy" Oct 30 22:37:04.944100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.943996 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bf96bc-6f37-4a91-aa39-4a988b797bd8" containerName="discovery" Oct 30 22:37:04.948657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.948632 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:04.998142 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:04.998109 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c29159\"/\"kube-root-ca.crt\"" Oct 30 22:37:05.020341 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.020306 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-home\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.020545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.020357 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-tmp-dir\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.020545 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.020478 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-dshm\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.020663 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.020634 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll6d9\" (UniqueName: \"kubernetes.io/projected/1d89bda7-e441-4fea-8c76-55de49833163-kube-api-access-ll6d9\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.020708 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.020691 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d89bda7-e441-4fea-8c76-55de49833163-tls-certs\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.020816 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.020800 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-model-cache\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.044653 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.044627 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c29159\"/\"openshift-service-ca.crt\"" Oct 30 22:37:05.044653 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.044642 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c29159\"/\"auth-test-model-c29159-kserve-self-signed-certs\"" Oct 30 22:37:05.044847 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.044723 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c29159\"/\"default-dockercfg-gpr5g\"" Oct 30 22:37:05.047122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.047100 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk"] Oct 30 22:37:05.122204 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122171 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll6d9\" (UniqueName: \"kubernetes.io/projected/1d89bda7-e441-4fea-8c76-55de49833163-kube-api-access-ll6d9\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.122388 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122218 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d89bda7-e441-4fea-8c76-55de49833163-tls-certs\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.122388 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122269 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-model-cache\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.122388 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122307 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-home\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.122604 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122464 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-tmp-dir\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.122604 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122531 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-dshm\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.122765 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122748 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-home\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.122859 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122844 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-model-cache\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.122951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.122857 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-tmp-dir\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.124804 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.124770 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-dshm\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.125062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.125047 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d89bda7-e441-4fea-8c76-55de49833163-tls-certs\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.180677 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.180644 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll6d9\" (UniqueName: \"kubernetes.io/projected/1d89bda7-e441-4fea-8c76-55de49833163-kube-api-access-ll6d9\") pod \"auth-test-model-c29159-kserve-74597cfb69-njfvk\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.260433 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.260381 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:05.635095 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.635013 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk"] Oct 30 22:37:05.641095 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:37:05.641067 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d89bda7_e441_4fea_8c76_55de49833163.slice/crio-0012e0993f0ecea45c5b73935f964351e20aeeca7d3d581f7edf0b2fe8997106 WatchSource:0}: Error finding container 0012e0993f0ecea45c5b73935f964351e20aeeca7d3d581f7edf0b2fe8997106: Status 404 returned error can't find the container with id 0012e0993f0ecea45c5b73935f964351e20aeeca7d3d581f7edf0b2fe8997106 Oct 30 22:37:05.869636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.869586 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" event={"ID":"1d89bda7-e441-4fea-8c76-55de49833163","Type":"ContainerStarted","Data":"fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a"} Oct 30 22:37:05.869808 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.869647 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" event={"ID":"1d89bda7-e441-4fea-8c76-55de49833163","Type":"ContainerStarted","Data":"0012e0993f0ecea45c5b73935f964351e20aeeca7d3d581f7edf0b2fe8997106"} Oct 30 22:37:05.981513 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:05.981405 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" podStartSLOduration=1.9813914129999999 podStartE2EDuration="1.981391413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:05.979678676 +0000 UTC m=+3534.570964163" watchObservedRunningTime="2026-10-30 22:37:05.981391413 +0000 UTC m=+3534.572676900" Oct 30 22:37:06.260901 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:06.260871 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:12.837440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.837392 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7"] Oct 30 22:37:12.837887 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.837874 5139 cpu_manager.go:414] "containerMap: removing container" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" Oct 30 22:37:12.837931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.837888 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" Oct 30 22:37:12.837998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.837989 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c79e9de-5f9e-497f-8892-1a7341d92672" containerName="istio-proxy" Oct 30 22:37:12.843199 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.843178 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:12.911634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.911601 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-tmp-dir\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:12.911819 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.911733 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-dshm\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:12.911819 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.911782 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-model-cache\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:12.911931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.911826 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-home\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:12.911931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.911919 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67adc476-875b-4005-bac9-5f5a22a94a11-tls-certs\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:12.912014 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.911982 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhm25\" (UniqueName: \"kubernetes.io/projected/67adc476-875b-4005-bac9-5f5a22a94a11-kube-api-access-jhm25\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:12.938804 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.938771 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-323082\"/\"openshift-service-ca.crt\"" Oct 30 22:37:12.939004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.938805 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-323082\"/\"kube-root-ca.crt\"" Oct 30 22:37:12.939004 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.938855 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-323082\"/\"auth-test-model-323082-kserve-self-signed-certs\"" Oct 30 22:37:12.939338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.939314 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-323082\"/\"default-dockercfg-n76s7\"" Oct 30 22:37:12.941555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:12.941533 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7"] Oct 30 22:37:13.012638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.012600 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-tmp-dir\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.012829 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.012715 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-dshm\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.012829 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.012747 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-model-cache\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.012963 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.012885 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-home\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.013028 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.012971 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-tmp-dir\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.013028 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.012986 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67adc476-875b-4005-bac9-5f5a22a94a11-tls-certs\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.013143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.013084 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-model-cache\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.013143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.013092 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhm25\" (UniqueName: \"kubernetes.io/projected/67adc476-875b-4005-bac9-5f5a22a94a11-kube-api-access-jhm25\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.013241 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.013187 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-home\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.015086 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.015060 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-dshm\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.015198 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.015180 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67adc476-875b-4005-bac9-5f5a22a94a11-tls-certs\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.071276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.071245 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhm25\" (UniqueName: \"kubernetes.io/projected/67adc476-875b-4005-bac9-5f5a22a94a11-kube-api-access-jhm25\") pod \"auth-test-model-323082-kserve-8688496f97-6m4m7\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.154726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.154661 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:13.528812 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.528782 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7"] Oct 30 22:37:13.535063 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:37:13.535032 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67adc476_875b_4005_bac9_5f5a22a94a11.slice/crio-0eee29ff8909745b2f1e73e2eb8c65de8bbd7e9e921a478f937af1bb55a386d7 WatchSource:0}: Error finding container 0eee29ff8909745b2f1e73e2eb8c65de8bbd7e9e921a478f937af1bb55a386d7: Status 404 returned error can't find the container with id 0eee29ff8909745b2f1e73e2eb8c65de8bbd7e9e921a478f937af1bb55a386d7 Oct 30 22:37:13.922301 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.922212 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" event={"ID":"67adc476-875b-4005-bac9-5f5a22a94a11","Type":"ContainerStarted","Data":"3988a17e77f878285373ca84ab97ec2d6981d09c6844084c4771fa40290ad4c5"} Oct 30 22:37:13.922301 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:13.922258 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" event={"ID":"67adc476-875b-4005-bac9-5f5a22a94a11","Type":"ContainerStarted","Data":"0eee29ff8909745b2f1e73e2eb8c65de8bbd7e9e921a478f937af1bb55a386d7"} Oct 30 22:37:14.036495 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:14.036446 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" podStartSLOduration=2.036432396 podStartE2EDuration="2.036432396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:14.035284047 +0000 UTC m=+3542.626569542" watchObservedRunningTime="2026-10-30 22:37:14.036432396 +0000 UTC m=+3542.627717877" Oct 30 22:37:14.154953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:14.154912 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:15.261236 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:15.261204 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:15.273902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:15.273876 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:15.944069 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:15.944039 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:37:19.804970 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:37:19.804926 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:37:23.155252 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:23.155216 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:23.168119 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:23.168093 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:23.985359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:23.985328 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:37:25.000908 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.000868 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft"] Oct 30 22:37:25.005178 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.005152 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.058064 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.058034 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-home\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.058214 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.058174 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxw5\" (UniqueName: \"kubernetes.io/projected/34142360-04b5-46f4-9f52-ba36d9ded3c9-kube-api-access-njxw5\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.058259 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.058240 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34142360-04b5-46f4-9f52-ba36d9ded3c9-tls-certs\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.058355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.058341 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-dshm\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.058440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.058371 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-tmp-dir\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.058510 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.058467 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-model-cache\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.100982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.100949 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-970415\"/\"kube-root-ca.crt\"" Oct 30 22:37:25.100982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.100961 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-970415\"/\"openshift-service-ca.crt\"" Oct 30 22:37:25.101180 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.101039 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-970415\"/\"noten-model-970415-kserve-self-signed-certs\"" Oct 30 22:37:25.101180 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.101091 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-970415\"/\"default-dockercfg-cgpj7\"" Oct 30 22:37:25.103568 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.103550 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft"] Oct 30 22:37:25.160432 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.160387 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxw5\" (UniqueName: \"kubernetes.io/projected/34142360-04b5-46f4-9f52-ba36d9ded3c9-kube-api-access-njxw5\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.160636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.160506 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34142360-04b5-46f4-9f52-ba36d9ded3c9-tls-certs\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.160636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.160565 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-dshm\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.160636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.160605 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-tmp-dir\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.160830 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.160648 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-model-cache\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.160999 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.160962 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-home\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.161161 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.161140 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-model-cache\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.161339 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.161313 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-tmp-dir\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.161449 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.161343 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-home\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.163391 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.163361 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-dshm\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.163636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.163619 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34142360-04b5-46f4-9f52-ba36d9ded3c9-tls-certs\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.218798 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.218754 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxw5\" (UniqueName: \"kubernetes.io/projected/34142360-04b5-46f4-9f52-ba36d9ded3c9-kube-api-access-njxw5\") pod \"noten-model-970415-kserve-d6b574469-9vjft\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.317721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.317649 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:25.691539 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.691467 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft"] Oct 30 22:37:25.698350 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:37:25.698312 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod34142360_04b5_46f4_9f52_ba36d9ded3c9.slice/crio-4025eac6333a4dd5f5ca28b54fb4b595a077bcecd29566f327ea6476dc9a006c WatchSource:0}: Error finding container 4025eac6333a4dd5f5ca28b54fb4b595a077bcecd29566f327ea6476dc9a006c: Status 404 returned error can't find the container with id 4025eac6333a4dd5f5ca28b54fb4b595a077bcecd29566f327ea6476dc9a006c Oct 30 22:37:25.988576 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.988497 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" event={"ID":"34142360-04b5-46f4-9f52-ba36d9ded3c9","Type":"ContainerStarted","Data":"450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9"} Oct 30 22:37:25.988576 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:25.988535 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" event={"ID":"34142360-04b5-46f4-9f52-ba36d9ded3c9","Type":"ContainerStarted","Data":"4025eac6333a4dd5f5ca28b54fb4b595a077bcecd29566f327ea6476dc9a006c"} Oct 30 22:37:26.099920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:26.099873 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" podStartSLOduration=2.099860112 podStartE2EDuration="2.099860112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:37:26.098355273 +0000 UTC m=+3554.689640760" watchObservedRunningTime="2026-10-30 22:37:26.099860112 +0000 UTC m=+3554.691145599" Oct 30 22:37:26.318028 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:26.317993 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:35.318797 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:35.318761 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:35.331582 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:35.331558 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:36.059383 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:36.059353 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:37:49.805222 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:37:49.805172 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:37:50.574225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:50.574192 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft"] Oct 30 22:37:50.755651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:50.755610 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq"] Oct 30 22:37:50.756062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:50.755855 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" podUID="82b42527-6fc7-4baf-8bc8-b0a494e11461" containerName="istio-proxy" containerID="cri-o://622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab" gracePeriod=30 Oct 30 22:37:55.567563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:55.567527 5139 patch_prober.go:29] interesting pod/e2e-noten-970415-openshift-default-658c8594d5-rddlq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:37:55.567965 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:55.567590 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" podUID="82b42527-6fc7-4baf-8bc8-b0a494e11461" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:37:56.028110 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.028084 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:37:56.108552 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108522 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-kube-api-access-wl7mk\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-kube-api-access-wl7mk\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.108688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108583 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-certs\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.108688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108645 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82b42527-6fc7-4baf-8bc8-b0a494e11461-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b42527-6fc7-4baf-8bc8-b0a494e11461-istiod-ca-cert\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.108688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108668 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-envoy\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.108852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108705 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-podinfo\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.108852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108734 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-token\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-token\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.108852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108845 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-credential-socket\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.109001 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108899 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-certs" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:56.109001 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108915 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-data\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.109001 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.108967 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-socket\") pod \"82b42527-6fc7-4baf-8bc8-b0a494e11461\" (UID: \"82b42527-6fc7-4baf-8bc8-b0a494e11461\") " Oct 30 22:37:56.109178 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109118 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82b42527-6fc7-4baf-8bc8-b0a494e11461-istiod-ca-cert" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:37:56.109178 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109145 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-credential-socket" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:56.109325 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109303 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-socket" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:56.109386 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109325 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-data" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:56.109581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109557 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.109657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109589 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.109657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109606 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.109657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109622 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.109657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.109637 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82b42527-6fc7-4baf-8bc8-b0a494e11461-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.110827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.110798 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-kube-api-access-wl7mk" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "kube-api-access-wl7mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:56.111003 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.110983 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-podinfo" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:37:56.111067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.111040 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-envoy" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:37:56.111371 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.111356 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-token" pod "82b42527-6fc7-4baf-8bc8-b0a494e11461" (UID: "82b42527-6fc7-4baf-8bc8-b0a494e11461"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:37:56.152900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.152826 5139 generic.go:356] "Generic (PLEG): container finished" podID="82b42527-6fc7-4baf-8bc8-b0a494e11461" containerID="622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab" exitCode=0 Oct 30 22:37:56.152900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.152869 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" Oct 30 22:37:56.153049 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.152912 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" event={"ID":"82b42527-6fc7-4baf-8bc8-b0a494e11461","Type":"ContainerDied","Data":"622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab"} Oct 30 22:37:56.153049 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.153001 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq" event={"ID":"82b42527-6fc7-4baf-8bc8-b0a494e11461","Type":"ContainerDied","Data":"d952a37da75d51a3a11888896754f77abc814543ca86b72a6ea42692a384d459"} Oct 30 22:37:56.153049 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.153024 5139 scope.go:122] "RemoveContainer" containerID="622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab" Oct 30 22:37:56.162716 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.162692 5139 scope.go:122] "RemoveContainer" containerID="622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab" Oct 30 22:37:56.162990 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:37:56.162972 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab\": container with ID starting with 622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab not found: ID does not exist" containerID="622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab" Oct 30 22:37:56.163043 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.162998 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab"} err="failed to get container status \"622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab\": rpc error: code = NotFound desc = could not find container \"622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab\": container with ID starting with 622775488b1c9179555cc0200b22dbd44e04825dc369f394de78604b757b82ab not found: ID does not exist" Oct 30 22:37:56.210434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.210388 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl7mk\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-kube-api-access-wl7mk\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.210434 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.210436 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.210673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.210448 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.210673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.210458 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82b42527-6fc7-4baf-8bc8-b0a494e11461-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:37:56.296950 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.296917 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq"] Oct 30 22:37:56.305054 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:56.305028 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-970415-openshift-default-658c8594d5-rddlq"] Oct 30 22:37:57.542525 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:37:57.542489 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82b42527-6fc7-4baf-8bc8-b0a494e11461" path="/var/lib/kubelet/pods/82b42527-6fc7-4baf-8bc8-b0a494e11461/volumes" Oct 30 22:38:05.595364 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:05.595305 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" podUID="34142360-04b5-46f4-9f52-ba36d9ded3c9" containerName="main" containerID="cri-o://450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9" gracePeriod=45 Oct 30 22:38:05.878058 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:05.878024 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:38:06.024110 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024069 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-tmp-dir\") pod \"34142360-04b5-46f4-9f52-ba36d9ded3c9\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " Oct 30 22:38:06.024314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024118 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34142360-04b5-46f4-9f52-ba36d9ded3c9-kube-api-access-njxw5\" (UniqueName: \"kubernetes.io/projected/34142360-04b5-46f4-9f52-ba36d9ded3c9-kube-api-access-njxw5\") pod \"34142360-04b5-46f4-9f52-ba36d9ded3c9\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " Oct 30 22:38:06.024314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024165 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-dshm\") pod \"34142360-04b5-46f4-9f52-ba36d9ded3c9\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " Oct 30 22:38:06.024314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024259 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34142360-04b5-46f4-9f52-ba36d9ded3c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/34142360-04b5-46f4-9f52-ba36d9ded3c9-tls-certs\") pod \"34142360-04b5-46f4-9f52-ba36d9ded3c9\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " Oct 30 22:38:06.024314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024293 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-home\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-home\") pod \"34142360-04b5-46f4-9f52-ba36d9ded3c9\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " Oct 30 22:38:06.024592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024346 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-tmp-dir" pod "34142360-04b5-46f4-9f52-ba36d9ded3c9" (UID: "34142360-04b5-46f4-9f52-ba36d9ded3c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:06.024592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024475 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-model-cache\") pod \"34142360-04b5-46f4-9f52-ba36d9ded3c9\" (UID: \"34142360-04b5-46f4-9f52-ba36d9ded3c9\") " Oct 30 22:38:06.024716 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024664 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-home" pod "34142360-04b5-46f4-9f52-ba36d9ded3c9" (UID: "34142360-04b5-46f4-9f52-ba36d9ded3c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:06.024823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024795 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-model-cache" pod "34142360-04b5-46f4-9f52-ba36d9ded3c9" (UID: "34142360-04b5-46f4-9f52-ba36d9ded3c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:06.024992 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.024976 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:06.025071 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.025000 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:06.025071 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.025012 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:06.026703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.026674 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34142360-04b5-46f4-9f52-ba36d9ded3c9-kube-api-access-njxw5" pod "34142360-04b5-46f4-9f52-ba36d9ded3c9" (UID: "34142360-04b5-46f4-9f52-ba36d9ded3c9"). InnerVolumeSpecName "kube-api-access-njxw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:06.026795 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.026695 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-dshm" pod "34142360-04b5-46f4-9f52-ba36d9ded3c9" (UID: "34142360-04b5-46f4-9f52-ba36d9ded3c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:06.026843 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.026807 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34142360-04b5-46f4-9f52-ba36d9ded3c9-tls-certs" pod "34142360-04b5-46f4-9f52-ba36d9ded3c9" (UID: "34142360-04b5-46f4-9f52-ba36d9ded3c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:38:06.125763 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.125681 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/34142360-04b5-46f4-9f52-ba36d9ded3c9-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:06.125763 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.125712 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxw5\" (UniqueName: \"kubernetes.io/projected/34142360-04b5-46f4-9f52-ba36d9ded3c9-kube-api-access-njxw5\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:06.125763 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.125723 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/34142360-04b5-46f4-9f52-ba36d9ded3c9-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:06.208815 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.208774 5139 generic.go:356] "Generic (PLEG): container finished" podID="34142360-04b5-46f4-9f52-ba36d9ded3c9" containerID="450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9" exitCode=0 Oct 30 22:38:06.208983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.208827 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" Oct 30 22:38:06.208983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.208826 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" event={"ID":"34142360-04b5-46f4-9f52-ba36d9ded3c9","Type":"ContainerDied","Data":"450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9"} Oct 30 22:38:06.208983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.208932 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft" event={"ID":"34142360-04b5-46f4-9f52-ba36d9ded3c9","Type":"ContainerDied","Data":"4025eac6333a4dd5f5ca28b54fb4b595a077bcecd29566f327ea6476dc9a006c"} Oct 30 22:38:06.208983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.208950 5139 scope.go:122] "RemoveContainer" containerID="450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9" Oct 30 22:38:06.219060 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.219039 5139 scope.go:122] "RemoveContainer" containerID="450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9" Oct 30 22:38:06.219365 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:06.219343 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9\": container with ID starting with 450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9 not found: ID does not exist" containerID="450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9" Oct 30 22:38:06.219408 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.219377 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9"} err="failed to get container status \"450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9\": rpc error: code = NotFound desc = could not find container \"450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9\": container with ID starting with 450f38d0165a6077bd19a223801f85185a6eb9972749f22c7616fcfdf0db04e9 not found: ID does not exist" Oct 30 22:38:06.366228 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.366195 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft"] Oct 30 22:38:06.370941 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:06.370909 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-970415/noten-model-970415-kserve-d6b574469-9vjft"] Oct 30 22:38:07.543287 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:07.543251 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34142360-04b5-46f4-9f52-ba36d9ded3c9" path="/var/lib/kubelet/pods/34142360-04b5-46f4-9f52-ba36d9ded3c9/volumes" Oct 30 22:38:14.603527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:14.603496 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt"] Oct 30 22:38:14.603963 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:14.603672 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" containerID="cri-o://9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316" gracePeriod=30 Oct 30 22:38:19.073483 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.073449 5139 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:38:19.073878 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.073508 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:38:19.808922 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:19.808864 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:38:19.865141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.865082 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:38:19.978524 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978493 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-token\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-token\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.978711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978541 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-socket\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.978711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978608 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-data\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.978711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978648 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-podinfo\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.978902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978713 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0fced4d-375e-4fa0-8d33-21329f3e7082-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fced4d-375e-4fa0-8d33-21329f3e7082-istiod-ca-cert\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.978902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978774 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-certs\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.978902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978851 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-kube-api-access-q9v27\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-kube-api-access-q9v27\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.979069 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978957 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-socket" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.979069 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.978972 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-data" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.979177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979058 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-certs" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.979177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979131 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-envoy\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.979177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979158 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fced4d-375e-4fa0-8d33-21329f3e7082-istiod-ca-cert" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:38:19.979177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979166 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-credential-socket\") pod \"c0fced4d-375e-4fa0-8d33-21329f3e7082\" (UID: \"c0fced4d-375e-4fa0-8d33-21329f3e7082\") " Oct 30 22:38:19.979486 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979455 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-credential-socket" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.979721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979700 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c0fced4d-375e-4fa0-8d33-21329f3e7082-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.979902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979881 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.980008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979907 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.980008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979922 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.980008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.979936 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:19.981026 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.981004 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-podinfo" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:38:19.981104 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.981025 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-token" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:19.981234 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.981215 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-envoy" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:38:19.981447 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:19.981410 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-kube-api-access-q9v27" pod "c0fced4d-375e-4fa0-8d33-21329f3e7082" (UID: "c0fced4d-375e-4fa0-8d33-21329f3e7082"). InnerVolumeSpecName "kube-api-access-q9v27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:20.081464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.081404 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9v27\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-kube-api-access-q9v27\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:20.081464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.081460 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:20.081867 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.081475 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:20.081867 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.081488 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c0fced4d-375e-4fa0-8d33-21329f3e7082-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:20.285273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.285240 5139 generic.go:356] "Generic (PLEG): container finished" podID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerID="9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316" exitCode=0 Oct 30 22:38:20.285455 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.285295 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" Oct 30 22:38:20.285455 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.285328 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" event={"ID":"c0fced4d-375e-4fa0-8d33-21329f3e7082","Type":"ContainerDied","Data":"9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316"} Oct 30 22:38:20.285455 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.285388 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt" event={"ID":"c0fced4d-375e-4fa0-8d33-21329f3e7082","Type":"ContainerDied","Data":"59443242768a047baf86dd3304e8450ae261f75b25bde3e4ae08d8172866f0ff"} Oct 30 22:38:20.285455 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.285434 5139 scope.go:122] "RemoveContainer" containerID="9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316" Oct 30 22:38:20.295109 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.295090 5139 scope.go:122] "RemoveContainer" containerID="9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316" Oct 30 22:38:20.295387 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:20.295362 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316\": container with ID starting with 9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316 not found: ID does not exist" containerID="9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316" Oct 30 22:38:20.295478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.295399 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316"} err="failed to get container status \"9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316\": rpc error: code = NotFound desc = could not find container \"9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316\": container with ID starting with 9e5f642307899d66763d3c3d65e2b4cd0583009b698cc6efb76bc54068164316 not found: ID does not exist" Oct 30 22:38:20.441100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.441062 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt"] Oct 30 22:38:20.487081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:20.487047 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-rv4nt"] Oct 30 22:38:21.543133 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:21.543051 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" path="/var/lib/kubelet/pods/c0fced4d-375e-4fa0-8d33-21329f3e7082/volumes" Oct 30 22:38:33.483983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.483947 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-453970"] Oct 30 22:38:33.484455 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484441 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" Oct 30 22:38:33.484455 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484456 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" Oct 30 22:38:33.484555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484467 5139 cpu_manager.go:414] "containerMap: removing container" podUID="34142360-04b5-46f4-9f52-ba36d9ded3c9" containerName="main" Oct 30 22:38:33.484555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484472 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34142360-04b5-46f4-9f52-ba36d9ded3c9" containerName="main" Oct 30 22:38:33.484555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484517 5139 cpu_manager.go:414] "containerMap: removing container" podUID="82b42527-6fc7-4baf-8bc8-b0a494e11461" containerName="istio-proxy" Oct 30 22:38:33.484555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484523 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82b42527-6fc7-4baf-8bc8-b0a494e11461" containerName="istio-proxy" Oct 30 22:38:33.484694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484589 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0fced4d-375e-4fa0-8d33-21329f3e7082" containerName="istio-proxy" Oct 30 22:38:33.484694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484599 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="34142360-04b5-46f4-9f52-ba36d9ded3c9" containerName="main" Oct 30 22:38:33.484694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.484611 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="82b42527-6fc7-4baf-8bc8-b0a494e11461" containerName="istio-proxy" Oct 30 22:38:33.489735 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.489712 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:33.539363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.539336 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jppg\"" Oct 30 22:38:33.587854 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.587808 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-453970"] Oct 30 22:38:33.635426 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.635386 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8zdk\" (UniqueName: \"kubernetes.io/projected/4ea9c4ae-788d-4040-95e0-ce0e50daf58f-kube-api-access-j8zdk\") pod \"test-curl-7664-453970\" (UID: \"4ea9c4ae-788d-4040-95e0-ce0e50daf58f\") " pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:33.737235 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.737130 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8zdk\" (UniqueName: \"kubernetes.io/projected/4ea9c4ae-788d-4040-95e0-ce0e50daf58f-kube-api-access-j8zdk\") pod \"test-curl-7664-453970\" (UID: \"4ea9c4ae-788d-4040-95e0-ce0e50daf58f\") " pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:33.798492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.798458 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8zdk\" (UniqueName: \"kubernetes.io/projected/4ea9c4ae-788d-4040-95e0-ce0e50daf58f-kube-api-access-j8zdk\") pod \"test-curl-7664-453970\" (UID: \"4ea9c4ae-788d-4040-95e0-ce0e50daf58f\") " pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:33.802377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:33.802357 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:34.176310 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:34.176274 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-453970"] Oct 30 22:38:34.181958 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:38:34.181931 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ea9c4ae_788d_4040_95e0_ce0e50daf58f.slice/crio-c2497957c27cd6f438e9cf6b37d4fd3dbde04012ae2191abb4662d405f3e35e0 WatchSource:0}: Error finding container c2497957c27cd6f438e9cf6b37d4fd3dbde04012ae2191abb4662d405f3e35e0: Status 404 returned error can't find the container with id c2497957c27cd6f438e9cf6b37d4fd3dbde04012ae2191abb4662d405f3e35e0 Oct 30 22:38:34.183885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:34.183868 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:38:34.362257 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:34.362215 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-453970" event={"ID":"4ea9c4ae-788d-4040-95e0-ce0e50daf58f","Type":"ContainerStarted","Data":"c2497957c27cd6f438e9cf6b37d4fd3dbde04012ae2191abb4662d405f3e35e0"} Oct 30 22:38:38.387335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:38.387242 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-453970" event={"ID":"4ea9c4ae-788d-4040-95e0-ce0e50daf58f","Type":"ContainerStarted","Data":"a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf"} Oct 30 22:38:38.496194 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:38.496144 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7664-453970" podStartSLOduration=1.563505401 podStartE2EDuration="5.496130381s" totalImagesPullingTime="3.93262498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:38.49462812 +0000 UTC m=+3627.085913609" watchObservedRunningTime="2026-10-30 22:38:38.496130381 +0000 UTC m=+3627.087415896" Oct 30 22:38:39.958888 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:39.958855 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-453970"] Oct 30 22:38:39.964861 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:39.964835 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-453970"] Oct 30 22:38:40.130447 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.130374 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7664-33eafa"] Oct 30 22:38:40.130916 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.130894 5139 cpu_manager.go:414] "containerMap: removing container" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" containerName="test-curl-7664-453970" Oct 30 22:38:40.130916 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.130912 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" containerName="test-curl-7664-453970" Oct 30 22:38:40.131085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.131030 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" containerName="test-curl-7664-453970" Oct 30 22:38:40.134352 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.134331 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:40.198228 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.198197 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-33eafa"] Oct 30 22:38:40.210754 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.210680 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lck2l\" (UniqueName: \"kubernetes.io/projected/8e42d81b-a25f-4a3a-abd4-d292f5651232-kube-api-access-lck2l\") pod \"test-curl-7664-33eafa\" (UID: \"8e42d81b-a25f-4a3a-abd4-d292f5651232\") " pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:40.311708 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.311675 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lck2l\" (UniqueName: \"kubernetes.io/projected/8e42d81b-a25f-4a3a-abd4-d292f5651232-kube-api-access-lck2l\") pod \"test-curl-7664-33eafa\" (UID: \"8e42d81b-a25f-4a3a-abd4-d292f5651232\") " pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:40.369564 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.369534 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lck2l\" (UniqueName: \"kubernetes.io/projected/8e42d81b-a25f-4a3a-abd4-d292f5651232-kube-api-access-lck2l\") pod \"test-curl-7664-33eafa\" (UID: \"8e42d81b-a25f-4a3a-abd4-d292f5651232\") " pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:40.398001 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.397962 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7664-453970" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" containerName="test-curl-7664-453970" containerID="cri-o://a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf" gracePeriod=2 Oct 30 22:38:40.445742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.445701 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:40.446046 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:40.446018 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-453970\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:40.815074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:40.814991 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7664-33eafa"] Oct 30 22:38:40.821010 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:38:40.820983 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e42d81b_a25f_4a3a_abd4_d292f5651232.slice/crio-aeffb4e0c58e086b23207f894fb38f9d8b7b3b971872709d07ca10019707f9f2 WatchSource:0}: Error finding container aeffb4e0c58e086b23207f894fb38f9d8b7b3b971872709d07ca10019707f9f2: Status 404 returned error can't find the container with id aeffb4e0c58e086b23207f894fb38f9d8b7b3b971872709d07ca10019707f9f2 Oct 30 22:38:41.404353 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:41.404317 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-33eafa" event={"ID":"8e42d81b-a25f-4a3a-abd4-d292f5651232","Type":"ContainerStarted","Data":"823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd"} Oct 30 22:38:41.404353 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:41.404360 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7664-33eafa" event={"ID":"8e42d81b-a25f-4a3a-abd4-d292f5651232","Type":"ContainerStarted","Data":"aeffb4e0c58e086b23207f894fb38f9d8b7b3b971872709d07ca10019707f9f2"} Oct 30 22:38:41.452182 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:41.452140 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-453970\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:41.562718 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:41.562664 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7664-33eafa" podStartSLOduration=1.5626461310000002 podStartE2EDuration="1.562646131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:41.559166816 +0000 UTC m=+3630.150452302" watchObservedRunningTime="2026-10-30 22:38:41.562646131 +0000 UTC m=+3630.153931620" Oct 30 22:38:42.640785 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:42.640757 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:42.688950 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:42.688872 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-453970\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:42.738712 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:42.738681 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ea9c4ae-788d-4040-95e0-ce0e50daf58f-kube-api-access-j8zdk\" (UniqueName: \"kubernetes.io/projected/4ea9c4ae-788d-4040-95e0-ce0e50daf58f-kube-api-access-j8zdk\") pod \"4ea9c4ae-788d-4040-95e0-ce0e50daf58f\" (UID: \"4ea9c4ae-788d-4040-95e0-ce0e50daf58f\") " Oct 30 22:38:42.740756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:42.740726 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ea9c4ae-788d-4040-95e0-ce0e50daf58f-kube-api-access-j8zdk" pod "4ea9c4ae-788d-4040-95e0-ce0e50daf58f" (UID: "4ea9c4ae-788d-4040-95e0-ce0e50daf58f"). InnerVolumeSpecName "kube-api-access-j8zdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:42.840181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:42.840150 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8zdk\" (UniqueName: \"kubernetes.io/projected/4ea9c4ae-788d-4040-95e0-ce0e50daf58f-kube-api-access-j8zdk\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:43.152883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.152849 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7664-33eafa"] Oct 30 22:38:43.157622 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.157593 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7664-33eafa"] Oct 30 22:38:43.418481 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.418381 5139 generic.go:356] "Generic (PLEG): container finished" podID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" containerID="a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf" exitCode=137 Oct 30 22:38:43.418481 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.418439 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:43.418703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.418479 5139 scope.go:122] "RemoveContainer" containerID="a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf" Oct 30 22:38:43.418703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.418651 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7664-33eafa" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" containerName="test-curl-7664-33eafa" containerID="cri-o://823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd" gracePeriod=2 Oct 30 22:38:43.427461 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.427436 5139 scope.go:122] "RemoveContainer" containerID="a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf" Oct 30 22:38:43.427741 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:43.427723 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf\": container with ID starting with a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf not found: ID does not exist" containerID="a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf" Oct 30 22:38:43.427805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.427749 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf"} err="failed to get container status \"a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf\": rpc error: code = NotFound desc = could not find container \"a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf\": container with ID starting with a9336744c2e4d413904a48e64a200088ce40f480752c769a8c83f67ca1257baf not found: ID does not exist" Oct 30 22:38:43.471868 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:43.471840 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-453970\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:43.519803 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:43.519775 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-453970\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" pod="opendatahub/test-curl-7664-453970" Oct 30 22:38:43.547030 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:43.546995 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ea9c4ae-788d-4040-95e0-ce0e50daf58f" path="/var/lib/kubelet/pods/4ea9c4ae-788d-4040-95e0-ce0e50daf58f/volumes" Oct 30 22:38:43.567704 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:43.567669 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-33eafa\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:44.489955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.489922 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc"] Oct 30 22:38:44.490482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.490461 5139 cpu_manager.go:414] "containerMap: removing container" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" containerName="test-curl-7664-33eafa" Oct 30 22:38:44.490562 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.490484 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" containerName="test-curl-7664-33eafa" Oct 30 22:38:44.490640 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.490629 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" containerName="test-curl-7664-33eafa" Oct 30 22:38:44.494312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.494290 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.546310 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.546278 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-03e614-openshift-default-dockercfg-m2l6k\"" Oct 30 22:38:44.598339 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.598305 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc"] Oct 30 22:38:44.666437 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666372 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-certs\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.666615 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666491 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s549\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-kube-api-access-5s549\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.666615 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666540 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-envoy\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.666740 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666681 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-socket\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.666803 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666745 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istiod-ca-cert\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.666803 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666784 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-token\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.666900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666865 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-credential-socket\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.666940 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666913 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-podinfo\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.666982 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.666965 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-data\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768308 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768218 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istiod-ca-cert\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768308 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768265 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-token\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768308 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768305 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-credential-socket\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768337 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-podinfo\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768400 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-data\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768616 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-certs\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768829 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768689 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s549\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-kube-api-access-5s549\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768829 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768739 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-envoy\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768829 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768810 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-data\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.768978 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768854 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-socket\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.769035 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.768996 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-certs\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.769138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.769110 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istiod-ca-cert\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.769334 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.769181 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-credential-socket\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.769438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.769243 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-socket\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.771158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.771134 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-envoy\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.771560 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.771537 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-podinfo\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.825918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.825893 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-token\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:44.826039 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:44.825933 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s549\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-kube-api-access-5s549\") pod \"e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:45.107214 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.107122 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:45.543440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.543379 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc"] Oct 30 22:38:45.547286 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:38:45.547261 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc489b5_736b_4a87_88fd_e74b5fe32eb2.slice/crio-964709dc0fc07e8c5ff468f50df9a057d83d200d9593345b021230d96cc94bd1 WatchSource:0}: Error finding container 964709dc0fc07e8c5ff468f50df9a057d83d200d9593345b021230d96cc94bd1: Status 404 returned error can't find the container with id 964709dc0fc07e8c5ff468f50df9a057d83d200d9593345b021230d96cc94bd1 Oct 30 22:38:45.550141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.550105 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:38:45.550231 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.550190 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:38:45.550269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.550231 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:38:45.642906 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.642881 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:45.679349 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.679262 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e42d81b-a25f-4a3a-abd4-d292f5651232-kube-api-access-lck2l\" (UniqueName: \"kubernetes.io/projected/8e42d81b-a25f-4a3a-abd4-d292f5651232-kube-api-access-lck2l\") pod \"8e42d81b-a25f-4a3a-abd4-d292f5651232\" (UID: \"8e42d81b-a25f-4a3a-abd4-d292f5651232\") " Oct 30 22:38:45.681300 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.681272 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e42d81b-a25f-4a3a-abd4-d292f5651232-kube-api-access-lck2l" pod "8e42d81b-a25f-4a3a-abd4-d292f5651232" (UID: "8e42d81b-a25f-4a3a-abd4-d292f5651232"). InnerVolumeSpecName "kube-api-access-lck2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:38:45.780548 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:45.780512 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lck2l\" (UniqueName: \"kubernetes.io/projected/8e42d81b-a25f-4a3a-abd4-d292f5651232-kube-api-access-lck2l\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:38:46.437352 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:46.437312 5139 generic.go:356] "Generic (PLEG): container finished" podID="8e42d81b-a25f-4a3a-abd4-d292f5651232" containerID="823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd" exitCode=137 Oct 30 22:38:46.437523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:46.437371 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:46.437523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:46.437404 5139 scope.go:122] "RemoveContainer" containerID="823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd" Oct 30 22:38:46.439335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:46.439309 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" event={"ID":"ecc489b5-736b-4a87-88fd-e74b5fe32eb2","Type":"ContainerStarted","Data":"5bd5e758acceea7eec90439315c7dff118f3c7575331456e4d6de1fb85f0a34a"} Oct 30 22:38:46.439464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:46.439360 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" event={"ID":"ecc489b5-736b-4a87-88fd-e74b5fe32eb2","Type":"ContainerStarted","Data":"964709dc0fc07e8c5ff468f50df9a057d83d200d9593345b021230d96cc94bd1"} Oct 30 22:38:46.446643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:46.446623 5139 scope.go:122] "RemoveContainer" containerID="823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd" Oct 30 22:38:46.446937 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:46.446918 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd\": container with ID starting with 823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd not found: ID does not exist" containerID="823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd" Oct 30 22:38:46.447008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:46.446945 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd"} err="failed to get container status \"823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd\": rpc error: code = NotFound desc = could not find container \"823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd\": container with ID starting with 823d14eef020b9a490c71945c232dbfabf37c7812a629120d03f9fb72d072dfd not found: ID does not exist" Oct 30 22:38:46.565526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:46.565466 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" podStartSLOduration=2.565452348 podStartE2EDuration="2.565452348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:46.56388296 +0000 UTC m=+3635.155168451" watchObservedRunningTime="2026-10-30 22:38:46.565452348 +0000 UTC m=+3635.156737835" Oct 30 22:38:46.611709 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:46.611671 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-33eafa\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:47.108177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:47.108142 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:47.113834 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:47.113808 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:47.275747 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:47.275705 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-33eafa\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:47.446998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:47.446918 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:47.447858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:47.447835 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:38:47.543685 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:47.543655 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" path="/var/lib/kubelet/pods/8e42d81b-a25f-4a3a-abd4-d292f5651232/volumes" Oct 30 22:38:47.610062 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:47.610018 5139 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7664-33eafa\" is forbidden: User \"system:node:ip-10-0-133-5.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-133-5.us-west-2.compute.internal' and this object" podUID="8e42d81b-a25f-4a3a-abd4-d292f5651232" pod="opendatahub/test-curl-7664-33eafa" Oct 30 22:38:49.804999 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:38:49.804966 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:38:53.210084 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.210049 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5"] Oct 30 22:38:53.216147 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.216125 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:53.270049 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.270013 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx48d\" (UniqueName: \"kubernetes.io/projected/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-kube-api-access-xx48d\") pod \"payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5\" (UID: \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:53.270355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.270289 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5\" (UID: \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:53.316482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.316454 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-03e614-dockercfg-8qxhv\"" Oct 30 22:38:53.316482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.316459 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-03e614\"" Oct 30 22:38:53.318706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.318686 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5"] Oct 30 22:38:53.371668 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.371641 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx48d\" (UniqueName: \"kubernetes.io/projected/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-kube-api-access-xx48d\") pod \"payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5\" (UID: \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:53.371826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.371810 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5\" (UID: \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:53.372534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.372516 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5\" (UID: \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:53.384583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.384557 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz"] Oct 30 22:38:53.391963 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.391935 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:53.429600 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.429576 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx48d\" (UniqueName: \"kubernetes.io/projected/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-kube-api-access-xx48d\") pod \"payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5\" (UID: \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:53.447167 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.447139 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz"] Oct 30 22:38:53.473108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.473032 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2gz\" (UniqueName: \"kubernetes.io/projected/c5b51f90-7eec-43b1-af98-3a40d6e66104-kube-api-access-vj2gz\") pod \"payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz\" (UID: \"c5b51f90-7eec-43b1-af98-3a40d6e66104\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:53.473246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.473170 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b51f90-7eec-43b1-af98-3a40d6e66104-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz\" (UID: \"c5b51f90-7eec-43b1-af98-3a40d6e66104\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:53.530362 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.530336 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:53.574735 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.574517 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b51f90-7eec-43b1-af98-3a40d6e66104-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz\" (UID: \"c5b51f90-7eec-43b1-af98-3a40d6e66104\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:53.575479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.575320 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b51f90-7eec-43b1-af98-3a40d6e66104-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz\" (UID: \"c5b51f90-7eec-43b1-af98-3a40d6e66104\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:53.575479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.575394 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2gz\" (UniqueName: \"kubernetes.io/projected/c5b51f90-7eec-43b1-af98-3a40d6e66104-kube-api-access-vj2gz\") pod \"payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz\" (UID: \"c5b51f90-7eec-43b1-af98-3a40d6e66104\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:53.633500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.633458 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2gz\" (UniqueName: \"kubernetes.io/projected/c5b51f90-7eec-43b1-af98-3a40d6e66104-kube-api-access-vj2gz\") pod \"payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz\" (UID: \"c5b51f90-7eec-43b1-af98-3a40d6e66104\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:53.707506 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.707473 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:53.906433 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:53.906393 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5"] Oct 30 22:38:53.913016 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:38:53.912988 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0828b8c5_62d8_48ac_b9ac_06dd7373aeb8.slice/crio-39c960906c7bdb96cbc96e2be3b4f28549c70de4371fa71c04d68c41389a8692 WatchSource:0}: Error finding container 39c960906c7bdb96cbc96e2be3b4f28549c70de4371fa71c04d68c41389a8692: Status 404 returned error can't find the container with id 39c960906c7bdb96cbc96e2be3b4f28549c70de4371fa71c04d68c41389a8692 Oct 30 22:38:54.082437 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.082347 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz"] Oct 30 22:38:54.087981 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:38:54.087943 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5b51f90_7eec_43b1_af98_3a40d6e66104.slice/crio-c1ec004dfd76e4e6dedfd87680de313aa576bb4b5e80e045b21a9078c5134c49 WatchSource:0}: Error finding container c1ec004dfd76e4e6dedfd87680de313aa576bb4b5e80e045b21a9078c5134c49: Status 404 returned error can't find the container with id c1ec004dfd76e4e6dedfd87680de313aa576bb4b5e80e045b21a9078c5134c49 Oct 30 22:38:54.487405 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.487311 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" event={"ID":"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8","Type":"ContainerStarted","Data":"70ec919562b3deb403b7eea992e65f2dd36039d5c5501a3a8949c4cfe259df00"} Oct 30 22:38:54.487405 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.487365 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" event={"ID":"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8","Type":"ContainerStarted","Data":"39c960906c7bdb96cbc96e2be3b4f28549c70de4371fa71c04d68c41389a8692"} Oct 30 22:38:54.487405 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.487394 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:38:54.489047 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.489023 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" event={"ID":"c5b51f90-7eec-43b1-af98-3a40d6e66104","Type":"ContainerStarted","Data":"56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa"} Oct 30 22:38:54.489144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.489064 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" event={"ID":"c5b51f90-7eec-43b1-af98-3a40d6e66104","Type":"ContainerStarted","Data":"c1ec004dfd76e4e6dedfd87680de313aa576bb4b5e80e045b21a9078c5134c49"} Oct 30 22:38:54.489144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.489088 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:38:54.597865 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.597811 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" podStartSLOduration=1.597794599 podStartE2EDuration="1.597794599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:54.596556616 +0000 UTC m=+3643.187842102" watchObservedRunningTime="2026-10-30 22:38:54.597794599 +0000 UTC m=+3643.189080091" Oct 30 22:38:54.707826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:38:54.707777 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" podStartSLOduration=1.7077633730000001 podStartE2EDuration="1.707763373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:38:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:38:54.705847899 +0000 UTC m=+3643.297133385" watchObservedRunningTime="2026-10-30 22:38:54.707763373 +0000 UTC m=+3643.299048860" Oct 30 22:39:05.497569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:05.497535 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:39:05.497981 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:05.497685 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:39:19.805546 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:39:19.805504 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:39:49.804703 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:39:49.804668 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:39:53.960428 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:53.960338 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47"] Oct 30 22:39:53.965534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:53.965508 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.014518 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.014487 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-03e614-serving-cert\"" Oct 30 22:39:54.046235 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.046204 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.046384 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.046306 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d291f9ba-26e8-45a5-86c1-91d9807c9e43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.046483 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.046456 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.046539 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.046522 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6xfg\" (UniqueName: \"kubernetes.io/projected/d291f9ba-26e8-45a5-86c1-91d9807c9e43-kube-api-access-x6xfg\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.065512 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.065482 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47"] Oct 30 22:39:54.148082 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.148047 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.148238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.148092 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6xfg\" (UniqueName: \"kubernetes.io/projected/d291f9ba-26e8-45a5-86c1-91d9807c9e43-kube-api-access-x6xfg\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.148238 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:39:54.148201 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-serving-cert: secret "maas-api-e2e-shared-a-w4-03e614-serving-cert" not found Oct 30 22:39:54.148335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.148209 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.148335 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:39:54.148279 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls podName:d291f9ba-26e8-45a5-86c1-91d9807c9e43 nodeName:}" failed. No retries permitted until 2026-10-30 22:39:54.648258951 +0000 UTC m=+3703.239544433 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls") pod "maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" (UID: "d291f9ba-26e8-45a5-86c1-91d9807c9e43") : secret "maas-api-e2e-shared-a-w4-03e614-serving-cert" not found Oct 30 22:39:54.148484 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.148339 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d291f9ba-26e8-45a5-86c1-91d9807c9e43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.148998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.148979 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d291f9ba-26e8-45a5-86c1-91d9807c9e43-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.150519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.150503 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.208364 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.208341 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6xfg\" (UniqueName: \"kubernetes.io/projected/d291f9ba-26e8-45a5-86c1-91d9807c9e43-kube-api-access-x6xfg\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.655009 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.654979 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.657355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.657334 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:54.878772 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:54.878741 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:39:55.249943 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:55.249914 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47"] Oct 30 22:39:55.257646 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:39:55.257615 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd291f9ba_26e8_45a5_86c1_91d9807c9e43.slice/crio-d46aabc54ff64c9c1ee871623667573925ee8de984385f94262f6bc5fb8ca395 WatchSource:0}: Error finding container d46aabc54ff64c9c1ee871623667573925ee8de984385f94262f6bc5fb8ca395: Status 404 returned error can't find the container with id d46aabc54ff64c9c1ee871623667573925ee8de984385f94262f6bc5fb8ca395 Oct 30 22:39:55.831220 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:55.831185 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" event={"ID":"d291f9ba-26e8-45a5-86c1-91d9807c9e43","Type":"ContainerStarted","Data":"d46aabc54ff64c9c1ee871623667573925ee8de984385f94262f6bc5fb8ca395"} Oct 30 22:39:56.840351 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:56.840313 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" event={"ID":"d291f9ba-26e8-45a5-86c1-91d9807c9e43","Type":"ContainerStarted","Data":"6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef"} Oct 30 22:39:56.840871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:39:56.840405 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:40:02.851886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:02.851853 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:40:02.965388 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:02.965330 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" podStartSLOduration=9.004757819 podStartE2EDuration="9.965316115s" totalImagesPullingTime="960.558296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:39:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:39:56.953346718 +0000 UTC m=+3705.544632205" watchObservedRunningTime="2026-10-30 22:40:02.965316115 +0000 UTC m=+3711.556601621" Oct 30 22:40:19.805099 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:40:19.805051 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:40:44.136108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:44.136070 5139 generic.go:356] "Generic (PLEG): container finished" podID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerID="5bd5e758acceea7eec90439315c7dff118f3c7575331456e4d6de1fb85f0a34a" exitCode=0 Oct 30 22:40:44.136703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:44.136147 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" event={"ID":"ecc489b5-736b-4a87-88fd-e74b5fe32eb2","Type":"ContainerDied","Data":"5bd5e758acceea7eec90439315c7dff118f3c7575331456e4d6de1fb85f0a34a"} Oct 30 22:40:44.136703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:44.136660 5139 scope.go:122] "RemoveContainer" containerID="5bd5e758acceea7eec90439315c7dff118f3c7575331456e4d6de1fb85f0a34a" Oct 30 22:40:44.138274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:44.138247 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:40:44.138340 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:44.138318 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:40:44.138377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:44.138349 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:40:45.144496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:45.144460 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" event={"ID":"ecc489b5-736b-4a87-88fd-e74b5fe32eb2","Type":"ContainerStarted","Data":"4585d23e0ea020e1b088397d97014911c9c3043ce0119ea76befbce3c6b6ca7c"} Oct 30 22:40:46.107670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:46.107631 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:40:46.108911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:46.108885 5139 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:40:46.109044 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:46.108931 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:40:47.107921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:47.107882 5139 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:40:47.108331 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:47.107936 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:40:47.446902 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:47.446813 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:40:48.108488 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:48.108454 5139 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:40:48.108924 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:48.108507 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:40:49.107769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:49.107738 5139 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:40:49.107951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:49.107790 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:40:49.817526 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:40:49.817496 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:40:50.108270 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:50.108139 5139 patch_prober.go:29] interesting pod/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 22:40:50.108270 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:50.108193 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 22:40:51.112469 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:51.112400 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:40:51.113509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:51.113487 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:40:52.440307 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:52.440266 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" podUID="3ab1a8fb7b394ed5c2ad3173d5294ab9" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:40:55.105343 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.105310 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk"] Oct 30 22:40:55.108668 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.108648 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.184663 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.184629 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-podinfo\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.184844 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.184683 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-data\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.184844 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.184750 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gfnl\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-kube-api-access-8gfnl\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.184844 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.184812 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-token\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.184952 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.184916 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-socket\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.185015 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.185002 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istiod-ca-cert\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.185054 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.185031 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-certs\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.185093 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.185074 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-credential-socket\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.185128 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.185095 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-envoy\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.210632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.210600 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk"] Oct 30 22:40:55.254835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.254800 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8c852b-openshift-default-dockercfg-brbjt\"" Oct 30 22:40:55.286230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286193 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-data\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286237 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gfnl\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-kube-api-access-8gfnl\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286285 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-token\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286336 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-socket\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286492 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istiod-ca-cert\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286529 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-certs\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286561 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-credential-socket\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286596 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-envoy\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286795 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286653 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-podinfo\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286795 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286683 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-data\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.286975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.286955 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-socket\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.287151 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.287029 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-credential-socket\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.287287 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.287047 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-certs\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.287590 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.287571 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istiod-ca-cert\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.289577 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.289552 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-envoy\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.289903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.289882 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-podinfo\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.346949 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.346915 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-token\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.347125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.347084 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gfnl\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-kube-api-access-8gfnl\") pod \"e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.421080 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.420993 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:55.854903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.854865 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk"] Oct 30 22:40:55.861332 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:40:55.861273 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92b6c51_1011_4fab_a6d3_929b0f849fc1.slice/crio-9b0c5872a16ce8a4f03cb21779a1c316c029370329469d25f524f125085684b3 WatchSource:0}: Error finding container 9b0c5872a16ce8a4f03cb21779a1c316c029370329469d25f524f125085684b3: Status 404 returned error can't find the container with id 9b0c5872a16ce8a4f03cb21779a1c316c029370329469d25f524f125085684b3 Oct 30 22:40:55.864262 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.864222 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:40:55.864372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.864316 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:40:55.864372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:55.864358 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:40:56.227808 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:56.227717 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" event={"ID":"a92b6c51-1011-4fab-a6d3-929b0f849fc1","Type":"ContainerStarted","Data":"0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658"} Oct 30 22:40:56.227808 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:56.227769 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" event={"ID":"a92b6c51-1011-4fab-a6d3-929b0f849fc1","Type":"ContainerStarted","Data":"9b0c5872a16ce8a4f03cb21779a1c316c029370329469d25f524f125085684b3"} Oct 30 22:40:56.343269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:56.343220 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" podStartSLOduration=1.34320641 podStartE2EDuration="1.34320641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:56.341718753 +0000 UTC m=+3764.933004240" watchObservedRunningTime="2026-10-30 22:40:56.34320641 +0000 UTC m=+3764.934491896" Oct 30 22:40:56.422058 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:56.422033 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:56.427798 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:56.427777 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:57.233095 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:57.233063 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:40:57.233870 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:40:57.233853 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:41:13.761747 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.761716 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp"] Oct 30 22:41:13.765017 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.764995 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797039 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797012 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-podinfo\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797054 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-token\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797136 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-kube-api-access-cxg2k\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797206 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-credential-socket\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797340 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797287 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-socket\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797340 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797321 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc58be14-16c1-4d36-a08e-9736d0c862dd-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797355 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-certs\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797384 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-envoy\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.797578 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.797521 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-data\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.869661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.869625 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp"] Oct 30 22:41:13.898746 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.898709 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-socket\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.898953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.898763 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc58be14-16c1-4d36-a08e-9736d0c862dd-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.898953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.898801 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-certs\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.898953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.898830 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-envoy\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.898953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.898925 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-data\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.899197 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899078 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-podinfo\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.899197 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899113 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-token\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.899197 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899145 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-socket\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.899197 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899167 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-kube-api-access-cxg2k\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.899197 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899171 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-certs\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.899661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899233 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-credential-socket\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.899661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899580 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-credential-socket\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.900009 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899808 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-data\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.900009 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.899894 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc58be14-16c1-4d36-a08e-9736d0c862dd-istiod-ca-cert\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.901784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.901764 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-envoy\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.901900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.901878 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-podinfo\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.912550 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.912531 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-2cc983-openshift-default-dockercfg-t52j7\"" Oct 30 22:41:13.956655 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.956626 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-token\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:13.956756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:13.956682 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-kube-api-access-cxg2k\") pod \"e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:14.077700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:14.077620 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:14.508478 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:14.508439 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp"] Oct 30 22:41:14.515823 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:41:14.515793 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc58be14_16c1_4d36_a08e_9736d0c862dd.slice/crio-ba28203d99468cd037550e17837e83393181835023403509bf9f30bf2962a63d WatchSource:0}: Error finding container ba28203d99468cd037550e17837e83393181835023403509bf9f30bf2962a63d: Status 404 returned error can't find the container with id ba28203d99468cd037550e17837e83393181835023403509bf9f30bf2962a63d Oct 30 22:41:14.518786 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:14.518749 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:41:14.518880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:14.518837 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:41:14.518936 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:14.518879 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:41:15.333120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:15.333082 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" event={"ID":"cc58be14-16c1-4d36-a08e-9736d0c862dd","Type":"ContainerStarted","Data":"fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec"} Oct 30 22:41:15.333539 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:15.333136 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" event={"ID":"cc58be14-16c1-4d36-a08e-9736d0c862dd","Type":"ContainerStarted","Data":"ba28203d99468cd037550e17837e83393181835023403509bf9f30bf2962a63d"} Oct 30 22:41:15.488050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:15.487990 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podStartSLOduration=2.487972112 podStartE2EDuration="2.487972112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:15.487689755 +0000 UTC m=+3784.078975241" watchObservedRunningTime="2026-10-30 22:41:15.487972112 +0000 UTC m=+3784.079257599" Oct 30 22:41:16.078636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:16.078603 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:16.084155 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:16.084126 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:16.177003 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:16.176973 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6"] Oct 30 22:41:16.342031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:16.341942 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:16.342922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:16.342901 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:41:19.804219 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:41:19.804180 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:41:28.387305 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.387228 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f"] Oct 30 22:41:28.390535 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.390511 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.441995 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.441912 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-1a5f4a-openshift-default-dockercfg-bw2gv\"" Oct 30 22:41:28.467030 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.466970 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-credential-socket\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.467490 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.467404 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-certs\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.467736 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.467717 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-socket\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.468121 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.468100 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-istio-token\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.468448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.468427 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20effb48-4149-4cca-893f-72b67c9cc66a-istiod-ca-cert\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.468801 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.468775 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20effb48-4149-4cca-893f-72b67c9cc66a-istio-podinfo\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.469150 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.469108 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-data\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.469583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.469504 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-envoy\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.469823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.469804 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w94h2\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-kube-api-access-w94h2\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.498335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.498307 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f"] Oct 30 22:41:28.571382 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571338 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-envoy\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.571585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571395 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w94h2\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-kube-api-access-w94h2\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.571585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571459 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-credential-socket\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.571585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571477 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-certs\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.571585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571513 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-socket\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.571585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571578 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-istio-token\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.571897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571610 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20effb48-4149-4cca-893f-72b67c9cc66a-istiod-ca-cert\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.571897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571662 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20effb48-4149-4cca-893f-72b67c9cc66a-istio-podinfo\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.571897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.571705 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-data\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.572088 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.572056 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-socket\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.572153 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.572065 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-data\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.572538 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.572512 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20effb48-4149-4cca-893f-72b67c9cc66a-istiod-ca-cert\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.572680 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.572568 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-credential-socket\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.572774 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.572610 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-certs\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.574443 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.574404 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-envoy\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.574745 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.574725 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20effb48-4149-4cca-893f-72b67c9cc66a-istio-podinfo\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.629901 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.629859 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w94h2\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-kube-api-access-w94h2\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.632912 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.632882 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-istio-token\") pod \"e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:28.704461 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:28.704370 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:29.129316 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:29.129282 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f"] Oct 30 22:41:29.134289 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:41:29.134259 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20effb48_4149_4cca_893f_72b67c9cc66a.slice/crio-5cacba1f9753ee9522396a19d9d30c8899092b5efa59de8f0c45a652b949d3f5 WatchSource:0}: Error finding container 5cacba1f9753ee9522396a19d9d30c8899092b5efa59de8f0c45a652b949d3f5: Status 404 returned error can't find the container with id 5cacba1f9753ee9522396a19d9d30c8899092b5efa59de8f0c45a652b949d3f5 Oct 30 22:41:29.137032 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:29.136997 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:41:29.137129 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:29.137065 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:41:29.137129 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:29.137098 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:41:29.418264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:29.418169 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" event={"ID":"20effb48-4149-4cca-893f-72b67c9cc66a","Type":"ContainerStarted","Data":"d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a"} Oct 30 22:41:29.418264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:29.418241 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" event={"ID":"20effb48-4149-4cca-893f-72b67c9cc66a","Type":"ContainerStarted","Data":"5cacba1f9753ee9522396a19d9d30c8899092b5efa59de8f0c45a652b949d3f5"} Oct 30 22:41:29.533994 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:29.533942 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" podStartSLOduration=1.53392791 podStartE2EDuration="1.53392791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:29.533505945 +0000 UTC m=+3798.124791433" watchObservedRunningTime="2026-10-30 22:41:29.53392791 +0000 UTC m=+3798.125213397" Oct 30 22:41:29.705102 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:29.705018 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:30.710711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:30.710683 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:31.197727 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.197680 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" podUID="f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" containerName="main" containerID="cri-o://910b73b90cfa452e5345016e3b856528c8651b9ef31ae3602605dd5dc1923d23" gracePeriod=45 Oct 30 22:41:31.432447 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.432398 5139 generic.go:356] "Generic (PLEG): container finished" podID="f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" containerID="910b73b90cfa452e5345016e3b856528c8651b9ef31ae3602605dd5dc1923d23" exitCode=0 Oct 30 22:41:31.432608 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.432475 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" event={"ID":"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4","Type":"ContainerDied","Data":"910b73b90cfa452e5345016e3b856528c8651b9ef31ae3602605dd5dc1923d23"} Oct 30 22:41:31.432763 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.432733 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:31.433690 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.433670 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:41:31.467338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.467320 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:41:31.608254 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.608221 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-dshm\") pod \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " Oct 30 22:41:31.608485 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.608326 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-kube-api-access-gj5pp\" (UniqueName: \"kubernetes.io/projected/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-kube-api-access-gj5pp\") pod \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " Oct 30 22:41:31.608575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.608496 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tls-certs\") pod \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " Oct 30 22:41:31.608575 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.608554 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-model-cache\") pod \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " Oct 30 22:41:31.608709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.608615 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tmp-dir\") pod \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " Oct 30 22:41:31.608709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.608651 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-home\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-home\") pod \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\" (UID: \"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4\") " Oct 30 22:41:31.612601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.609868 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-model-cache" pod "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" (UID: "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.612601 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.610121 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tmp-dir" pod "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" (UID: "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.612822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.612611 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-home" pod "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" (UID: "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.614123 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.614101 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-dshm" pod "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" (UID: "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:41:31.617270 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.617249 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tls-certs" pod "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" (UID: "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:41:31.617357 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.617324 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-kube-api-access-gj5pp" pod "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" (UID: "f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4"). InnerVolumeSpecName "kube-api-access-gj5pp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:41:31.713482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.713389 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.713482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.713434 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.713482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.713447 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.713482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.713459 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.713482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.713472 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:31.713482 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:31.713484 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj5pp\" (UniqueName: \"kubernetes.io/projected/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4-kube-api-access-gj5pp\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:41:32.439618 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:32.439591 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" Oct 30 22:41:32.439806 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:32.439627 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6" event={"ID":"f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4","Type":"ContainerDied","Data":"9c09451cf75d82580e3a3e8539b1a370325ed11d74b8ba1699988a6e8d29b139"} Oct 30 22:41:32.439806 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:32.439680 5139 scope.go:122] "RemoveContainer" containerID="910b73b90cfa452e5345016e3b856528c8651b9ef31ae3602605dd5dc1923d23" Oct 30 22:41:32.594857 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:32.594827 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6"] Oct 30 22:41:32.640933 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:32.640903 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-ae7a1f/auth-test-model-ae7a1f-kserve-78dc648477-xz2t6"] Oct 30 22:41:33.544332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:41:33.544295 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" path="/var/lib/kubelet/pods/f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4/volumes" Oct 30 22:41:49.803574 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:41:49.803540 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:42:10.676901 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.676870 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv"] Oct 30 22:42:10.677367 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.677354 5139 cpu_manager.go:414] "containerMap: removing container" podUID="f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" containerName="main" Oct 30 22:42:10.677367 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.677368 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" containerName="main" Oct 30 22:42:10.677504 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.677493 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dc40bf-9912-49ff-a7eb-4ef8e6cddaa4" containerName="main" Oct 30 22:42:10.681325 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.681306 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.730389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.730363 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-8c852b-serving-cert\"" Oct 30 22:42:10.740807 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.740785 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv"] Oct 30 22:42:10.768174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.768128 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa931b0-386c-44c1-91bd-b6ec8940ad26-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.768326 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.768261 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.768326 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.768319 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shfhz\" (UniqueName: \"kubernetes.io/projected/5fa931b0-386c-44c1-91bd-b6ec8940ad26-kube-api-access-shfhz\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.768410 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.768343 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.869510 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.869479 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa931b0-386c-44c1-91bd-b6ec8940ad26-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.869688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.869533 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.869688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.869571 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shfhz\" (UniqueName: \"kubernetes.io/projected/5fa931b0-386c-44c1-91bd-b6ec8940ad26-kube-api-access-shfhz\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.869688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.869592 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.869865 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:42:10.869689 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-serving-cert: secret "maas-api-e2e-extmod-a-8c852b-serving-cert" not found Oct 30 22:42:10.869865 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:42:10.869744 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls podName:5fa931b0-386c-44c1-91bd-b6ec8940ad26 nodeName:}" failed. No retries permitted until 2026-10-30 22:42:11.369727873 +0000 UTC m=+3839.961013356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls") pod "maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" (UID: "5fa931b0-386c-44c1-91bd-b6ec8940ad26") : secret "maas-api-e2e-extmod-a-8c852b-serving-cert" not found Oct 30 22:42:10.870137 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.870115 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa931b0-386c-44c1-91bd-b6ec8940ad26-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.871958 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.871936 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-metrics-certs\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:10.928058 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:10.927989 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shfhz\" (UniqueName: \"kubernetes.io/projected/5fa931b0-386c-44c1-91bd-b6ec8940ad26-kube-api-access-shfhz\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:11.375830 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:11.375794 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:11.378231 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:11.378211 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls\") pod \"maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:11.594570 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:11.594539 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:11.974554 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:11.974478 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv"] Oct 30 22:42:11.980575 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:42:11.980550 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa931b0_386c_44c1_91bd_b6ec8940ad26.slice/crio-318319e8844030621f1928305ae13b30818edd868cd72dd581007befa8192d9b WatchSource:0}: Error finding container 318319e8844030621f1928305ae13b30818edd868cd72dd581007befa8192d9b: Status 404 returned error can't find the container with id 318319e8844030621f1928305ae13b30818edd868cd72dd581007befa8192d9b Oct 30 22:42:12.677896 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:12.677861 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" event={"ID":"5fa931b0-386c-44c1-91bd-b6ec8940ad26","Type":"ContainerStarted","Data":"318319e8844030621f1928305ae13b30818edd868cd72dd581007befa8192d9b"} Oct 30 22:42:13.685599 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:13.685561 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" event={"ID":"5fa931b0-386c-44c1-91bd-b6ec8940ad26","Type":"ContainerStarted","Data":"cb322c90ff860d63f7058f9635ec0c17640afaa15fa50ecb2652e6b2d0f647f1"} Oct 30 22:42:13.686032 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:13.685751 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:13.801033 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:13.800977 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" podStartSLOduration=2.900795096 podStartE2EDuration="3.800960081s" totalImagesPullingTime="900.164985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:13.799292698 +0000 UTC m=+3842.390578184" watchObservedRunningTime="2026-10-30 22:42:13.800960081 +0000 UTC m=+3842.392245567" Oct 30 22:42:19.698291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:19.698258 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:42:19.805311 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:42:19.805278 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:42:33.806254 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.806211 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd"] Oct 30 22:42:33.810832 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.810807 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.861688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.861638 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.861885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.861723 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpfd6\" (UniqueName: \"kubernetes.io/projected/772c18d4-c0b2-4aa7-952d-25e11e4a945f-kube-api-access-vpfd6\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.861963 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.861886 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.861963 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.861937 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772c18d4-c0b2-4aa7-952d-25e11e4a945f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.862289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.862269 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-2cc983-serving-cert\"" Oct 30 22:42:33.911497 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.911388 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd"] Oct 30 22:42:33.962904 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.962858 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772c18d4-c0b2-4aa7-952d-25e11e4a945f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.963074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.963008 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.963074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.963049 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpfd6\" (UniqueName: \"kubernetes.io/projected/772c18d4-c0b2-4aa7-952d-25e11e4a945f-kube-api-access-vpfd6\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.963180 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.963105 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.963230 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:42:33.963220 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-serving-cert: secret "maas-api-e2e-shared-b-w4-2cc983-serving-cert" not found Oct 30 22:42:33.963307 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:42:33.963289 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls podName:772c18d4-c0b2-4aa7-952d-25e11e4a945f nodeName:}" failed. No retries permitted until 2026-10-30 22:42:34.463269044 +0000 UTC m=+3863.054554525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls") pod "maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" (UID: "772c18d4-c0b2-4aa7-952d-25e11e4a945f") : secret "maas-api-e2e-shared-b-w4-2cc983-serving-cert" not found Oct 30 22:42:33.963699 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.963674 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772c18d4-c0b2-4aa7-952d-25e11e4a945f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:33.965791 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:33.965770 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:34.022099 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:34.022068 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpfd6\" (UniqueName: \"kubernetes.io/projected/772c18d4-c0b2-4aa7-952d-25e11e4a945f-kube-api-access-vpfd6\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:34.467935 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:34.467903 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:34.470267 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:34.470249 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:34.723528 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:34.723456 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:35.107517 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:35.107472 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd"] Oct 30 22:42:35.114988 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:42:35.114962 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772c18d4_c0b2_4aa7_952d_25e11e4a945f.slice/crio-c15075c23754fc0fe921e5abd9e9087cae85c979146e0060f6cba545b770149d WatchSource:0}: Error finding container c15075c23754fc0fe921e5abd9e9087cae85c979146e0060f6cba545b770149d: Status 404 returned error can't find the container with id c15075c23754fc0fe921e5abd9e9087cae85c979146e0060f6cba545b770149d Oct 30 22:42:35.810799 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:35.810764 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" event={"ID":"772c18d4-c0b2-4aa7-952d-25e11e4a945f","Type":"ContainerStarted","Data":"c15075c23754fc0fe921e5abd9e9087cae85c979146e0060f6cba545b770149d"} Oct 30 22:42:36.818210 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:36.818168 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" event={"ID":"772c18d4-c0b2-4aa7-952d-25e11e4a945f","Type":"ContainerStarted","Data":"3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96"} Oct 30 22:42:36.818652 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:36.818364 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:36.934885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:36.934832 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" podStartSLOduration=3.120424701 podStartE2EDuration="3.934818219s" totalImagesPullingTime="814.393518ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:36.933015548 +0000 UTC m=+3865.524301035" watchObservedRunningTime="2026-10-30 22:42:36.934818219 +0000 UTC m=+3865.526103742" Oct 30 22:42:42.825506 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:42.825462 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.136:8443/health\": dial tcp 10.134.0.136:8443: connect: connection refused" Oct 30 22:42:46.877303 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:46.877269 5139 generic.go:356] "Generic (PLEG): container finished" podID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerID="3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96" exitCode=1 Oct 30 22:42:46.877787 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:46.877317 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" event={"ID":"772c18d4-c0b2-4aa7-952d-25e11e4a945f","Type":"ContainerDied","Data":"3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96"} Oct 30 22:42:46.877787 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:46.877711 5139 scope.go:122] "RemoveContainer" containerID="3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96" Oct 30 22:42:47.884389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:47.884357 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" event={"ID":"772c18d4-c0b2-4aa7-952d-25e11e4a945f","Type":"ContainerStarted","Data":"7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff"} Oct 30 22:42:47.884846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:47.884582 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:42:49.804933 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:42:49.804898 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:42:51.717965 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:51.717880 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7"] Oct 30 22:42:52.440427 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:52.440377 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" podUID="3ab1a8fb7b394ed5c2ad3173d5294ab9" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded" Oct 30 22:42:53.914188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:42:53.913034 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:43:01.838522 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:01.838112 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": read tcp 10.134.0.2:57752->10.134.0.133:15021: read: connection reset by peer" start-of-body= Oct 30 22:43:01.838522 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:01.838177 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": read tcp 10.134.0.2:57752->10.134.0.133:15021: read: connection reset by peer" Oct 30 22:43:01.977931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:01.977897 5139 generic.go:356] "Generic (PLEG): container finished" podID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerID="fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec" exitCode=0 Oct 30 22:43:01.978122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:01.977977 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" event={"ID":"cc58be14-16c1-4d36-a08e-9736d0c862dd","Type":"ContainerDied","Data":"fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec"} Oct 30 22:43:01.978596 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:01.978576 5139 scope.go:122] "RemoveContainer" containerID="fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec" Oct 30 22:43:01.980650 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:01.980612 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:01.980751 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:01.980707 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:01.980810 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:01.980782 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:03.000442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:02.998098 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" event={"ID":"cc58be14-16c1-4d36-a08e-9736d0c862dd","Type":"ContainerStarted","Data":"fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d"} Oct 30 22:43:03.004436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.001547 5139 generic.go:356] "Generic (PLEG): container finished" podID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerID="0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658" exitCode=0 Oct 30 22:43:03.004436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.001618 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" event={"ID":"a92b6c51-1011-4fab-a6d3-929b0f849fc1","Type":"ContainerDied","Data":"0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658"} Oct 30 22:43:03.004436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.002087 5139 scope.go:122] "RemoveContainer" containerID="0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658" Oct 30 22:43:03.004436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.004357 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:03.005191 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.004474 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:03.005191 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.004527 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:03.078651 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.078609 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:43:03.081217 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.081171 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:43:03.081470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:03.081374 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 22:43:04.023023 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:04.022991 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" event={"ID":"a92b6c51-1011-4fab-a6d3-929b0f849fc1","Type":"ContainerStarted","Data":"faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60"} Oct 30 22:43:04.082441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:04.080820 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:43:04.082441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:04.080891 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 22:43:04.421312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:04.421217 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:43:04.421885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:04.421857 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:43:04.422084 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:04.422060 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 22:43:05.078809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:05.078774 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:43:05.079432 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:05.079382 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 22:43:05.421510 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:05.421429 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:43:05.421510 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:05.421480 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 22:43:06.078816 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:06.078777 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:43:06.079470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:06.079437 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 22:43:06.424535 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:06.424439 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:43:06.424535 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:06.424494 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 22:43:06.739058 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:06.738962 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" podUID="67adc476-875b-4005-bac9-5f5a22a94a11" containerName="main" containerID="cri-o://3988a17e77f878285373ca84ab97ec2d6981d09c6844084c4771fa40290ad4c5" gracePeriod=45 Oct 30 22:43:07.043387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.043351 5139 generic.go:356] "Generic (PLEG): container finished" podID="67adc476-875b-4005-bac9-5f5a22a94a11" containerID="3988a17e77f878285373ca84ab97ec2d6981d09c6844084c4771fa40290ad4c5" exitCode=0 Oct 30 22:43:07.043615 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.043559 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" event={"ID":"67adc476-875b-4005-bac9-5f5a22a94a11","Type":"ContainerDied","Data":"3988a17e77f878285373ca84ab97ec2d6981d09c6844084c4771fa40290ad4c5"} Oct 30 22:43:07.087258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.087222 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:43:07.095555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.095513 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 22:43:07.142466 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.142443 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:43:07.254661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.254624 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67adc476-875b-4005-bac9-5f5a22a94a11-kube-api-access-jhm25\" (UniqueName: \"kubernetes.io/projected/67adc476-875b-4005-bac9-5f5a22a94a11-kube-api-access-jhm25\") pod \"67adc476-875b-4005-bac9-5f5a22a94a11\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " Oct 30 22:43:07.254661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.254668 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-tmp-dir\") pod \"67adc476-875b-4005-bac9-5f5a22a94a11\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " Oct 30 22:43:07.254915 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.254739 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-home\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-home\") pod \"67adc476-875b-4005-bac9-5f5a22a94a11\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " Oct 30 22:43:07.254981 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.254933 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-tmp-dir" pod "67adc476-875b-4005-bac9-5f5a22a94a11" (UID: "67adc476-875b-4005-bac9-5f5a22a94a11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:07.255038 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.254986 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-home" pod "67adc476-875b-4005-bac9-5f5a22a94a11" (UID: "67adc476-875b-4005-bac9-5f5a22a94a11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:07.255038 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.255021 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-dshm\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-dshm\") pod \"67adc476-875b-4005-bac9-5f5a22a94a11\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " Oct 30 22:43:07.255145 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.255109 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-model-cache\") pod \"67adc476-875b-4005-bac9-5f5a22a94a11\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " Oct 30 22:43:07.255201 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.255170 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67adc476-875b-4005-bac9-5f5a22a94a11-tls-certs\" (UniqueName: \"kubernetes.io/secret/67adc476-875b-4005-bac9-5f5a22a94a11-tls-certs\") pod \"67adc476-875b-4005-bac9-5f5a22a94a11\" (UID: \"67adc476-875b-4005-bac9-5f5a22a94a11\") " Oct 30 22:43:07.255392 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.255361 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-model-cache" pod "67adc476-875b-4005-bac9-5f5a22a94a11" (UID: "67adc476-875b-4005-bac9-5f5a22a94a11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:07.255654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.255634 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.255751 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.255659 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.255751 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.255671 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.257577 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.257532 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67adc476-875b-4005-bac9-5f5a22a94a11-tls-certs" pod "67adc476-875b-4005-bac9-5f5a22a94a11" (UID: "67adc476-875b-4005-bac9-5f5a22a94a11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:07.259069 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.259032 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-dshm" pod "67adc476-875b-4005-bac9-5f5a22a94a11" (UID: "67adc476-875b-4005-bac9-5f5a22a94a11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:07.259275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.259254 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67adc476-875b-4005-bac9-5f5a22a94a11-kube-api-access-jhm25" pod "67adc476-875b-4005-bac9-5f5a22a94a11" (UID: "67adc476-875b-4005-bac9-5f5a22a94a11"). InnerVolumeSpecName "kube-api-access-jhm25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:07.356742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.356658 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67adc476-875b-4005-bac9-5f5a22a94a11-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.356742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.356692 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhm25\" (UniqueName: \"kubernetes.io/projected/67adc476-875b-4005-bac9-5f5a22a94a11-kube-api-access-jhm25\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.356742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.356703 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67adc476-875b-4005-bac9-5f5a22a94a11-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:07.422175 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.422140 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:43:07.422351 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.422190 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 22:43:07.981441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:07.978499 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" podUID="67adc476-875b-4005-bac9-5f5a22a94a11" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:43:08.051477 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.051042 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" Oct 30 22:43:08.051477 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.051106 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7" event={"ID":"67adc476-875b-4005-bac9-5f5a22a94a11","Type":"ContainerDied","Data":"0eee29ff8909745b2f1e73e2eb8c65de8bbd7e9e921a478f937af1bb55a386d7"} Oct 30 22:43:08.051477 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.051146 5139 scope.go:122] "RemoveContainer" containerID="3988a17e77f878285373ca84ab97ec2d6981d09c6844084c4771fa40290ad4c5" Oct 30 22:43:08.083439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.080748 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 30 22:43:08.083439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.080813 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 30 22:43:08.214119 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.213628 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7"] Oct 30 22:43:08.215338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.215316 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-323082/auth-test-model-323082-kserve-8688496f97-6m4m7"] Oct 30 22:43:08.421671 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.421636 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 30 22:43:08.421851 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:08.421685 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 30 22:43:09.091089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:09.091052 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:43:09.091475 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:09.091454 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:43:09.092360 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:09.092338 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:43:09.426108 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:09.426023 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:43:09.426595 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:09.426407 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:43:09.427036 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:09.427017 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:43:09.545263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:09.545225 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67adc476-875b-4005-bac9-5f5a22a94a11" path="/var/lib/kubelet/pods/67adc476-875b-4005-bac9-5f5a22a94a11/volumes" Oct 30 22:43:19.805242 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:43:19.805205 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:43:36.235831 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:36.235754 5139 generic.go:356] "Generic (PLEG): container finished" podID="20effb48-4149-4cca-893f-72b67c9cc66a" containerID="d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a" exitCode=0 Oct 30 22:43:36.236245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:36.235828 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" event={"ID":"20effb48-4149-4cca-893f-72b67c9cc66a","Type":"ContainerDied","Data":"d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a"} Oct 30 22:43:36.236245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:36.236228 5139 scope.go:122] "RemoveContainer" containerID="d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a" Oct 30 22:43:36.237799 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:36.237766 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:36.237916 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:36.237847 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:36.237916 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:36.237898 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:37.243264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:37.243226 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" event={"ID":"20effb48-4149-4cca-893f-72b67c9cc66a","Type":"ContainerStarted","Data":"e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387"} Oct 30 22:43:37.704796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:37.704699 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:43:37.704984 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:37.704880 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:43:37.704984 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:37.704933 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:43:38.705174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:38.705133 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:43:38.705834 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:38.705798 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:43:39.704960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:39.704924 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:43:39.705131 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:39.704982 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:43:40.704867 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:40.704834 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:43:40.705277 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:40.704898 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:43:41.705730 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:41.705692 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:43:41.706114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:41.705755 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:43:42.709199 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:42.709171 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:43:42.709694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:42.709598 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:43:42.710234 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:42.710216 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:43:43.568536 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:43.568506 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk"] Oct 30 22:43:45.333832 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.333802 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk"] Oct 30 22:43:45.334330 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.334316 5139 cpu_manager.go:414] "containerMap: removing container" podUID="67adc476-875b-4005-bac9-5f5a22a94a11" containerName="main" Oct 30 22:43:45.334376 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.334331 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67adc476-875b-4005-bac9-5f5a22a94a11" containerName="main" Oct 30 22:43:45.334468 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.334457 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="67adc476-875b-4005-bac9-5f5a22a94a11" containerName="main" Oct 30 22:43:45.337464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.337443 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383136 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-certs\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383176 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-token\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383210 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-credential-socket\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383452 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383372 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqfht\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-kube-api-access-sqfht\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383452 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383429 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-socket\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383557 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383461 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-envoy\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383557 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383502 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-podinfo\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383569 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istiod-ca-cert\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.383626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.383602 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-data\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.415946 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.415914 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk"] Oct 30 22:43:45.432570 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.432545 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-5a5534-openshift-default-dockercfg-mtk8k\"" Oct 30 22:43:45.485188 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485156 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-data\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485401 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485296 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-certs\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485401 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485343 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-token\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485401 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485377 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-credential-socket\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485461 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqfht\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-kube-api-access-sqfht\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485491 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-socket\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485521 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-envoy\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485562 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-podinfo\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485598 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istiod-ca-cert\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485664 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-data\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.485903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485727 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-certs\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.486019 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485940 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-socket\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.486019 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.485994 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-credential-socket\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.486559 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.486531 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istiod-ca-cert\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.488371 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.488350 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-envoy\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.488560 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.488537 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-podinfo\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.543454 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.543396 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-token\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.543774 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.543751 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqfht\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-kube-api-access-sqfht\") pod \"e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:45.649735 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:45.649653 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:46.078866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:46.078835 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk"] Oct 30 22:43:46.084841 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:43:46.084812 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e8093e_00ea_4e68_8923_d3556d1ae1e3.slice/crio-6ae7608276d9528e4f35980f6821244865b275d42c2d70d94a36c678509583d3 WatchSource:0}: Error finding container 6ae7608276d9528e4f35980f6821244865b275d42c2d70d94a36c678509583d3: Status 404 returned error can't find the container with id 6ae7608276d9528e4f35980f6821244865b275d42c2d70d94a36c678509583d3 Oct 30 22:43:46.087607 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:46.087576 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:46.087667 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:46.087643 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:46.087707 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:46.087675 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:43:46.319255 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:46.319222 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" event={"ID":"51e8093e-00ea-4e68-8923-d3556d1ae1e3","Type":"ContainerStarted","Data":"0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f"} Oct 30 22:43:46.319449 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:46.319277 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" event={"ID":"51e8093e-00ea-4e68-8923-d3556d1ae1e3","Type":"ContainerStarted","Data":"6ae7608276d9528e4f35980f6821244865b275d42c2d70d94a36c678509583d3"} Oct 30 22:43:46.435809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:46.435698 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" podStartSLOduration=1.435676722 podStartE2EDuration="1.435676722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:46.433189702 +0000 UTC m=+3935.024475200" watchObservedRunningTime="2026-10-30 22:43:46.435676722 +0000 UTC m=+3935.026962210" Oct 30 22:43:46.649749 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:46.649722 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:47.656292 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:47.656260 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:48.331438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:48.331374 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:48.332343 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:48.332326 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:43:49.805183 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:43:49.805144 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:43:58.589354 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.589300 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" podUID="1d89bda7-e441-4fea-8c76-55de49833163" containerName="main" containerID="cri-o://fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a" gracePeriod=45 Oct 30 22:43:58.854508 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.854452 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:43:58.948031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948000 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d89bda7-e441-4fea-8c76-55de49833163-tls-certs\" (UniqueName: \"kubernetes.io/secret/1d89bda7-e441-4fea-8c76-55de49833163-tls-certs\") pod \"1d89bda7-e441-4fea-8c76-55de49833163\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " Oct 30 22:43:58.948203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948053 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-home\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-home\") pod \"1d89bda7-e441-4fea-8c76-55de49833163\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " Oct 30 22:43:58.948203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948115 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-model-cache\") pod \"1d89bda7-e441-4fea-8c76-55de49833163\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " Oct 30 22:43:58.948341 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948275 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-tmp-dir\") pod \"1d89bda7-e441-4fea-8c76-55de49833163\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " Oct 30 22:43:58.948341 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948316 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-dshm\") pod \"1d89bda7-e441-4fea-8c76-55de49833163\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " Oct 30 22:43:58.948496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948327 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-home" pod "1d89bda7-e441-4fea-8c76-55de49833163" (UID: "1d89bda7-e441-4fea-8c76-55de49833163"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:58.948496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948365 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d89bda7-e441-4fea-8c76-55de49833163-kube-api-access-ll6d9\" (UniqueName: \"kubernetes.io/projected/1d89bda7-e441-4fea-8c76-55de49833163-kube-api-access-ll6d9\") pod \"1d89bda7-e441-4fea-8c76-55de49833163\" (UID: \"1d89bda7-e441-4fea-8c76-55de49833163\") " Oct 30 22:43:58.948496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948403 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-model-cache" pod "1d89bda7-e441-4fea-8c76-55de49833163" (UID: "1d89bda7-e441-4fea-8c76-55de49833163"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:58.948640 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948496 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-tmp-dir" pod "1d89bda7-e441-4fea-8c76-55de49833163" (UID: "1d89bda7-e441-4fea-8c76-55de49833163"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:58.948881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948855 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:58.948881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948882 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:58.949078 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.948895 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:58.950410 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.950383 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-dshm" pod "1d89bda7-e441-4fea-8c76-55de49833163" (UID: "1d89bda7-e441-4fea-8c76-55de49833163"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:43:58.950410 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.950444 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d89bda7-e441-4fea-8c76-55de49833163-tls-certs" pod "1d89bda7-e441-4fea-8c76-55de49833163" (UID: "1d89bda7-e441-4fea-8c76-55de49833163"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:43:58.950604 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:58.950510 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d89bda7-e441-4fea-8c76-55de49833163-kube-api-access-ll6d9" pod "1d89bda7-e441-4fea-8c76-55de49833163" (UID: "1d89bda7-e441-4fea-8c76-55de49833163"). InnerVolumeSpecName "kube-api-access-ll6d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:43:59.050519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.050485 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d89bda7-e441-4fea-8c76-55de49833163-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:59.050519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.050513 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d89bda7-e441-4fea-8c76-55de49833163-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:59.050712 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.050528 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ll6d9\" (UniqueName: \"kubernetes.io/projected/1d89bda7-e441-4fea-8c76-55de49833163-kube-api-access-ll6d9\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:43:59.394424 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.394385 5139 generic.go:356] "Generic (PLEG): container finished" podID="1d89bda7-e441-4fea-8c76-55de49833163" containerID="fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a" exitCode=0 Oct 30 22:43:59.394591 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.394448 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" Oct 30 22:43:59.394591 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.394466 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" event={"ID":"1d89bda7-e441-4fea-8c76-55de49833163","Type":"ContainerDied","Data":"fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a"} Oct 30 22:43:59.394591 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.394516 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk" event={"ID":"1d89bda7-e441-4fea-8c76-55de49833163","Type":"ContainerDied","Data":"0012e0993f0ecea45c5b73935f964351e20aeeca7d3d581f7edf0b2fe8997106"} Oct 30 22:43:59.394591 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.394540 5139 scope.go:122] "RemoveContainer" containerID="fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a" Oct 30 22:43:59.404857 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.404839 5139 scope.go:122] "RemoveContainer" containerID="fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a" Oct 30 22:43:59.405138 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:43:59.405115 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a\": container with ID starting with fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a not found: ID does not exist" containerID="fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a" Oct 30 22:43:59.405214 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.405149 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a"} err="failed to get container status \"fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a\": rpc error: code = NotFound desc = could not find container \"fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a\": container with ID starting with fb805d3417a3ecca285313f2e3cc2606e193e76fc3655c3cc6abcc11e633836a not found: ID does not exist" Oct 30 22:43:59.549910 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.549880 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk"] Oct 30 22:43:59.596402 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:43:59.596374 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c29159/auth-test-model-c29159-kserve-74597cfb69-njfvk"] Oct 30 22:44:01.543385 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:01.543349 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d89bda7-e441-4fea-8c76-55de49833163" path="/var/lib/kubelet/pods/1d89bda7-e441-4fea-8c76-55de49833163/volumes" Oct 30 22:44:10.027945 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.027909 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882"] Oct 30 22:44:10.028487 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.028472 5139 cpu_manager.go:414] "containerMap: removing container" podUID="1d89bda7-e441-4fea-8c76-55de49833163" containerName="main" Oct 30 22:44:10.028541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.028488 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d89bda7-e441-4fea-8c76-55de49833163" containerName="main" Oct 30 22:44:10.028590 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.028581 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d89bda7-e441-4fea-8c76-55de49833163" containerName="main" Oct 30 22:44:10.032403 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.032373 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.100263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.100220 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4f44d4-a36e-4779-b7ab-908329d4691b-istiod-ca-cert\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.100656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.100635 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-data\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.100852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.100834 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-credential-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.101010 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.100995 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-certs\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.101183 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.101168 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-envoy\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.101366 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.101351 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-token\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.101563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.101545 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.101719 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.101704 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlzww\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-kube-api-access-dlzww\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.101885 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.101871 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-podinfo\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.114667 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.114638 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6e0b99-openshift-default-dockercfg-pcv5l\"" Oct 30 22:44:10.141475 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.141373 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882"] Oct 30 22:44:10.203637 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.203595 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4f44d4-a36e-4779-b7ab-908329d4691b-istiod-ca-cert\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.203828 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.203713 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-data\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.203828 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.203750 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-credential-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.203828 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.203775 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-certs\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.203989 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.203844 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-envoy\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.203989 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.203914 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-token\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.203989 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.203954 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.203989 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.203983 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlzww\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-kube-api-access-dlzww\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.204181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.204044 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-podinfo\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.204181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.204136 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-credential-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.204519 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.204494 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4f44d4-a36e-4779-b7ab-908329d4691b-istiod-ca-cert\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.204719 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.204546 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-data\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.204816 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.204603 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-certs\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.204816 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.204657 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.206895 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.206847 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-envoy\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.207122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.207103 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-podinfo\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.262763 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.262728 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-token\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.263002 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.262981 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlzww\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-kube-api-access-dlzww\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.345487 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.345365 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:10.782876 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.782842 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882"] Oct 30 22:44:10.786785 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:44:10.786753 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4f44d4_a36e_4779_b7ab_908329d4691b.slice/crio-e83d8bcc3daf8f046201f7742f562ec21d10784e50fbb0fc69b89e6827a160dd WatchSource:0}: Error finding container e83d8bcc3daf8f046201f7742f562ec21d10784e50fbb0fc69b89e6827a160dd: Status 404 returned error can't find the container with id e83d8bcc3daf8f046201f7742f562ec21d10784e50fbb0fc69b89e6827a160dd Oct 30 22:44:10.789472 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.789438 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:10.789557 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.789506 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:10.789557 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:10.789536 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:44:11.480826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:11.480780 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" event={"ID":"3c4f44d4-a36e-4779-b7ab-908329d4691b","Type":"ContainerStarted","Data":"d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5"} Oct 30 22:44:11.481239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:11.480852 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" event={"ID":"3c4f44d4-a36e-4779-b7ab-908329d4691b","Type":"ContainerStarted","Data":"e83d8bcc3daf8f046201f7742f562ec21d10784e50fbb0fc69b89e6827a160dd"} Oct 30 22:44:11.643295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:11.643244 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" podStartSLOduration=2.6432306 podStartE2EDuration="2.6432306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:11.641328106 +0000 UTC m=+3960.232613603" watchObservedRunningTime="2026-10-30 22:44:11.6432306 +0000 UTC m=+3960.234516137" Oct 30 22:44:12.346065 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:12.346034 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:12.352109 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:12.352086 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:12.487302 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:12.487266 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:12.488003 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:12.487982 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:44:19.805664 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:44:19.805628 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:44:24.994307 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:24.994233 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn"] Oct 30 22:44:24.997731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:24.997707 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.047191 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.047165 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-be6ecc-serving-cert\"" Oct 30 22:44:25.092595 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.092558 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-metrics-certs\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.092754 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.092733 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc4g9\" (UniqueName: \"kubernetes.io/projected/d7aedde2-8cea-4844-a490-01605c800cbe-kube-api-access-sc4g9\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.092822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.092771 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.092822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.092796 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aedde2-8cea-4844-a490-01605c800cbe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.101271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.101247 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn"] Oct 30 22:44:25.194497 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.194461 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-metrics-certs\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.194735 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.194594 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc4g9\" (UniqueName: \"kubernetes.io/projected/d7aedde2-8cea-4844-a490-01605c800cbe-kube-api-access-sc4g9\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.194735 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.194622 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.194735 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.194639 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aedde2-8cea-4844-a490-01605c800cbe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.194915 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:44:25.194764 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-serving-cert: secret "maas-api-e2e-extmod-a-be6ecc-serving-cert" not found Oct 30 22:44:25.194915 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:44:25.194823 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls podName:d7aedde2-8cea-4844-a490-01605c800cbe nodeName:}" failed. No retries permitted until 2026-10-30 22:44:25.694809404 +0000 UTC m=+3974.286094873 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls") pod "maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" (UID: "d7aedde2-8cea-4844-a490-01605c800cbe") : secret "maas-api-e2e-extmod-a-be6ecc-serving-cert" not found Oct 30 22:44:25.195304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.195281 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aedde2-8cea-4844-a490-01605c800cbe-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.196833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.196815 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-metrics-certs\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.253955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.253888 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc4g9\" (UniqueName: \"kubernetes.io/projected/d7aedde2-8cea-4844-a490-01605c800cbe-kube-api-access-sc4g9\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.701658 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.701575 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.704074 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.704047 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls\") pod \"maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:25.910124 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:25.910087 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:26.286103 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:26.286074 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn"] Oct 30 22:44:26.292404 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:44:26.292368 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7aedde2_8cea_4844_a490_01605c800cbe.slice/crio-13aee7d3cb57e6c1ad33bd7042c109a953173dbd8aced1a2b0b5ee0d57d5076f WatchSource:0}: Error finding container 13aee7d3cb57e6c1ad33bd7042c109a953173dbd8aced1a2b0b5ee0d57d5076f: Status 404 returned error can't find the container with id 13aee7d3cb57e6c1ad33bd7042c109a953173dbd8aced1a2b0b5ee0d57d5076f Oct 30 22:44:26.293882 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:26.293862 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:44:26.616757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:26.616671 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" event={"ID":"d7aedde2-8cea-4844-a490-01605c800cbe","Type":"ContainerStarted","Data":"13aee7d3cb57e6c1ad33bd7042c109a953173dbd8aced1a2b0b5ee0d57d5076f"} Oct 30 22:44:27.629684 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:27.629636 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" event={"ID":"d7aedde2-8cea-4844-a490-01605c800cbe","Type":"ContainerStarted","Data":"299ec7e776d7d3ba9a5548c8d7bad5c655cbb34891a3ccb83e3bc92763cc3c07"} Oct 30 22:44:27.743066 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:27.743018 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" podStartSLOduration=2.868195269 podStartE2EDuration="3.743006267s" totalImagesPullingTime="874.810998ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:27.742086975 +0000 UTC m=+3976.333372453" watchObservedRunningTime="2026-10-30 22:44:27.743006267 +0000 UTC m=+3976.334291754" Oct 30 22:44:28.635469 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:28.635408 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:34.647429 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:44:34.647382 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:44:49.807504 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:44:49.807465 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:45:00.284592 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.284555 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6"] Oct 30 22:45:00.288677 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.288655 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:00.301966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.301940 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf"] Oct 30 22:45:00.305236 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.305217 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:00.313262 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.313239 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdp6g\" (UniqueName: \"kubernetes.io/projected/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-kube-api-access-cdp6g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf\" (UID: \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:00.313361 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.313316 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6\" (UID: \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:00.313403 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.313383 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf\" (UID: \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:00.313464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.313452 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhrx5\" (UniqueName: \"kubernetes.io/projected/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-kube-api-access-xhrx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6\" (UID: \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:00.320195 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.320172 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg"] Oct 30 22:45:00.322944 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.322924 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:00.369464 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.369407 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\"" Oct 30 22:45:00.411211 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.411165 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6"] Oct 30 22:45:00.414777 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.414751 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf"] Oct 30 22:45:00.415660 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.415630 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6\" (UID: \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:00.415797 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.415690 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf\" (UID: \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:00.415797 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.415752 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c061-9ced-41a5-9764-e087e66668a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg\" (UID: \"ef28c061-9ced-41a5-9764-e087e66668a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:00.415797 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.415784 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhrx5\" (UniqueName: \"kubernetes.io/projected/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-kube-api-access-xhrx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6\" (UID: \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:00.415990 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.415946 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/ef28c061-9ced-41a5-9764-e087e66668a9-kube-api-access-dq7b4\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg\" (UID: \"ef28c061-9ced-41a5-9764-e087e66668a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:00.416050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.415991 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdp6g\" (UniqueName: \"kubernetes.io/projected/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-kube-api-access-cdp6g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf\" (UID: \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:00.416697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.416625 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf\" (UID: \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:00.417326 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.417300 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6\" (UID: \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:00.418205 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.418184 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg"] Oct 30 22:45:00.473882 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.473848 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdp6g\" (UniqueName: \"kubernetes.io/projected/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-kube-api-access-cdp6g\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf\" (UID: \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:00.474812 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.474791 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhrx5\" (UniqueName: \"kubernetes.io/projected/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-kube-api-access-xhrx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6\" (UID: \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:00.517196 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.517164 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c061-9ced-41a5-9764-e087e66668a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg\" (UID: \"ef28c061-9ced-41a5-9764-e087e66668a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:00.517372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.517265 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/ef28c061-9ced-41a5-9764-e087e66668a9-kube-api-access-dq7b4\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg\" (UID: \"ef28c061-9ced-41a5-9764-e087e66668a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:00.518000 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.517977 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c061-9ced-41a5-9764-e087e66668a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg\" (UID: \"ef28c061-9ced-41a5-9764-e087e66668a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:00.575118 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.575052 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/ef28c061-9ced-41a5-9764-e087e66668a9-kube-api-access-dq7b4\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg\" (UID: \"ef28c061-9ced-41a5-9764-e087e66668a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:00.601095 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.601065 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:00.617113 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.617089 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:00.634826 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.634803 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:00.983355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:00.983281 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6"] Oct 30 22:45:00.989156 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:45:00.989127 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfb611b7_5ee2_4456_9520_e8fbc2f5f80c.slice/crio-20362f5a48ab906ddd0d71716607a8e20bb4dabd56dba3097dc9d85213a8c13e WatchSource:0}: Error finding container 20362f5a48ab906ddd0d71716607a8e20bb4dabd56dba3097dc9d85213a8c13e: Status 404 returned error can't find the container with id 20362f5a48ab906ddd0d71716607a8e20bb4dabd56dba3097dc9d85213a8c13e Oct 30 22:45:01.035135 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.035107 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf"] Oct 30 22:45:01.038009 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.037982 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg"] Oct 30 22:45:01.046194 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:45:01.046166 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef28c061_9ced_41a5_9764_e087e66668a9.slice/crio-2f021dd53434dce047e0dbbfe0af84e7825a9e188253b5a92295c18ca3b19f74 WatchSource:0}: Error finding container 2f021dd53434dce047e0dbbfe0af84e7825a9e188253b5a92295c18ca3b19f74: Status 404 returned error can't find the container with id 2f021dd53434dce047e0dbbfe0af84e7825a9e188253b5a92295c18ca3b19f74 Oct 30 22:45:01.829635 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.829587 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" event={"ID":"ef28c061-9ced-41a5-9764-e087e66668a9","Type":"ContainerStarted","Data":"48a86c8773c6cd61644ea377a7ccf417c09921a5cf6b897cd00de092711854bb"} Oct 30 22:45:01.830057 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.829648 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" event={"ID":"ef28c061-9ced-41a5-9764-e087e66668a9","Type":"ContainerStarted","Data":"2f021dd53434dce047e0dbbfe0af84e7825a9e188253b5a92295c18ca3b19f74"} Oct 30 22:45:01.831148 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.831118 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" event={"ID":"9d469dac-2a7b-4e33-bd97-9a17b664e1a1","Type":"ContainerStarted","Data":"33e8f04df15c9021c7d78bda9a8eb720538f33d9399600587b85d54ab4503884"} Oct 30 22:45:01.831274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.831157 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" event={"ID":"9d469dac-2a7b-4e33-bd97-9a17b664e1a1","Type":"ContainerStarted","Data":"40e5ce63ab219d7093d2643f74c0675446ff5b6227965530daa8e32116a4b57e"} Oct 30 22:45:01.832793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.832756 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" event={"ID":"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c","Type":"ContainerStarted","Data":"62713fcc7390575f27aeaa265805a27d0459a66a2c76fceb96cd5a30ffcd8f26"} Oct 30 22:45:01.832918 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.832809 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" event={"ID":"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c","Type":"ContainerStarted","Data":"20362f5a48ab906ddd0d71716607a8e20bb4dabd56dba3097dc9d85213a8c13e"} Oct 30 22:45:01.940776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:01.940728 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" podStartSLOduration=1.9407146100000001 podStartE2EDuration="1.94071461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:01.938242493 +0000 UTC m=+4010.529527981" watchObservedRunningTime="2026-10-30 22:45:01.94071461 +0000 UTC m=+4010.532000129" Oct 30 22:45:02.056986 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.056931 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" podStartSLOduration=2.056916702 podStartE2EDuration="2.056916702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.055533074 +0000 UTC m=+4010.646818561" watchObservedRunningTime="2026-10-30 22:45:02.056916702 +0000 UTC m=+4010.648202189" Oct 30 22:45:02.167857 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.167800 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" podStartSLOduration=2.167786245 podStartE2EDuration="2.167786245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.165193391 +0000 UTC m=+4010.756478877" watchObservedRunningTime="2026-10-30 22:45:02.167786245 +0000 UTC m=+4010.759071732" Oct 30 22:45:02.840073 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.840038 5139 generic.go:356] "Generic (PLEG): container finished" podID="ef28c061-9ced-41a5-9764-e087e66668a9" containerID="48a86c8773c6cd61644ea377a7ccf417c09921a5cf6b897cd00de092711854bb" exitCode=60 Oct 30 22:45:02.840565 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.840121 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" event={"ID":"ef28c061-9ced-41a5-9764-e087e66668a9","Type":"ContainerDied","Data":"48a86c8773c6cd61644ea377a7ccf417c09921a5cf6b897cd00de092711854bb"} Oct 30 22:45:02.840565 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.840442 5139 scope.go:122] "RemoveContainer" containerID="48a86c8773c6cd61644ea377a7ccf417c09921a5cf6b897cd00de092711854bb" Oct 30 22:45:02.841800 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.841776 5139 generic.go:356] "Generic (PLEG): container finished" podID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerID="33e8f04df15c9021c7d78bda9a8eb720538f33d9399600587b85d54ab4503884" exitCode=60 Oct 30 22:45:02.841907 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.841848 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" event={"ID":"9d469dac-2a7b-4e33-bd97-9a17b664e1a1","Type":"ContainerDied","Data":"33e8f04df15c9021c7d78bda9a8eb720538f33d9399600587b85d54ab4503884"} Oct 30 22:45:02.842084 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.842067 5139 scope.go:122] "RemoveContainer" containerID="33e8f04df15c9021c7d78bda9a8eb720538f33d9399600587b85d54ab4503884" Oct 30 22:45:02.843741 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.843561 5139 generic.go:356] "Generic (PLEG): container finished" podID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerID="62713fcc7390575f27aeaa265805a27d0459a66a2c76fceb96cd5a30ffcd8f26" exitCode=60 Oct 30 22:45:02.843741 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.843624 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" event={"ID":"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c","Type":"ContainerDied","Data":"62713fcc7390575f27aeaa265805a27d0459a66a2c76fceb96cd5a30ffcd8f26"} Oct 30 22:45:02.843850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:02.843802 5139 scope.go:122] "RemoveContainer" containerID="62713fcc7390575f27aeaa265805a27d0459a66a2c76fceb96cd5a30ffcd8f26" Oct 30 22:45:03.852134 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:03.852100 5139 generic.go:356] "Generic (PLEG): container finished" podID="ef28c061-9ced-41a5-9764-e087e66668a9" containerID="ce939d9d89231dc4f802990d5a43521425d734ff18a70444e0932b9caee1f82d" exitCode=60 Oct 30 22:45:03.852583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:03.852165 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" event={"ID":"ef28c061-9ced-41a5-9764-e087e66668a9","Type":"ContainerDied","Data":"ce939d9d89231dc4f802990d5a43521425d734ff18a70444e0932b9caee1f82d"} Oct 30 22:45:03.852583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:03.852229 5139 scope.go:122] "RemoveContainer" containerID="48a86c8773c6cd61644ea377a7ccf417c09921a5cf6b897cd00de092711854bb" Oct 30 22:45:03.852583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:03.852524 5139 scope.go:122] "RemoveContainer" containerID="ce939d9d89231dc4f802990d5a43521425d734ff18a70444e0932b9caee1f82d" Oct 30 22:45:03.852925 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:03.852900 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg_odh-ai-gateway-infra(ef28c061-9ced-41a5-9764-e087e66668a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" Oct 30 22:45:03.854468 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:03.854446 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" event={"ID":"9d469dac-2a7b-4e33-bd97-9a17b664e1a1","Type":"ContainerStarted","Data":"4b08448519186ea76b66ebd7dbe62c09026f1d9c474d2740200712abf00c763a"} Oct 30 22:45:03.856361 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:03.856337 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" event={"ID":"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c","Type":"ContainerStarted","Data":"c49d2be05d972c5f6050dbd307e51df2e68e8e0c83d8ee15b9af34b86e83756f"} Oct 30 22:45:04.866372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.866333 5139 scope.go:122] "RemoveContainer" containerID="ce939d9d89231dc4f802990d5a43521425d734ff18a70444e0932b9caee1f82d" Oct 30 22:45:04.866978 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:04.866720 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg_odh-ai-gateway-infra(ef28c061-9ced-41a5-9764-e087e66668a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" Oct 30 22:45:04.868332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.868303 5139 generic.go:356] "Generic (PLEG): container finished" podID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerID="4b08448519186ea76b66ebd7dbe62c09026f1d9c474d2740200712abf00c763a" exitCode=60 Oct 30 22:45:04.868477 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.868390 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" event={"ID":"9d469dac-2a7b-4e33-bd97-9a17b664e1a1","Type":"ContainerDied","Data":"4b08448519186ea76b66ebd7dbe62c09026f1d9c474d2740200712abf00c763a"} Oct 30 22:45:04.868562 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.868507 5139 scope.go:122] "RemoveContainer" containerID="33e8f04df15c9021c7d78bda9a8eb720538f33d9399600587b85d54ab4503884" Oct 30 22:45:04.868701 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.868679 5139 scope.go:122] "RemoveContainer" containerID="4b08448519186ea76b66ebd7dbe62c09026f1d9c474d2740200712abf00c763a" Oct 30 22:45:04.869071 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:04.869042 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf_odh-ai-gateway-infra(9d469dac-2a7b-4e33-bd97-9a17b664e1a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" Oct 30 22:45:04.870796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.870775 5139 generic.go:356] "Generic (PLEG): container finished" podID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerID="c49d2be05d972c5f6050dbd307e51df2e68e8e0c83d8ee15b9af34b86e83756f" exitCode=60 Oct 30 22:45:04.870893 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.870852 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" event={"ID":"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c","Type":"ContainerDied","Data":"c49d2be05d972c5f6050dbd307e51df2e68e8e0c83d8ee15b9af34b86e83756f"} Oct 30 22:45:04.871069 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.871055 5139 scope.go:122] "RemoveContainer" containerID="c49d2be05d972c5f6050dbd307e51df2e68e8e0c83d8ee15b9af34b86e83756f" Oct 30 22:45:04.871373 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:04.871354 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6_odh-ai-gateway-infra(cfb611b7-5ee2-4456-9520-e8fbc2f5f80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" Oct 30 22:45:04.880096 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:04.880068 5139 scope.go:122] "RemoveContainer" containerID="62713fcc7390575f27aeaa265805a27d0459a66a2c76fceb96cd5a30ffcd8f26" Oct 30 22:45:05.879015 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:05.878979 5139 scope.go:122] "RemoveContainer" containerID="4b08448519186ea76b66ebd7dbe62c09026f1d9c474d2740200712abf00c763a" Oct 30 22:45:05.879465 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:05.879337 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf_odh-ai-gateway-infra(9d469dac-2a7b-4e33-bd97-9a17b664e1a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" Oct 30 22:45:05.880479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:05.880461 5139 scope.go:122] "RemoveContainer" containerID="c49d2be05d972c5f6050dbd307e51df2e68e8e0c83d8ee15b9af34b86e83756f" Oct 30 22:45:05.880739 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:05.880719 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6_odh-ai-gateway-infra(cfb611b7-5ee2-4456-9520-e8fbc2f5f80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" Oct 30 22:45:14.536359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:14.536330 5139 scope.go:122] "RemoveContainer" containerID="ce939d9d89231dc4f802990d5a43521425d734ff18a70444e0932b9caee1f82d" Oct 30 22:45:15.536523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:15.536494 5139 scope.go:122] "RemoveContainer" containerID="c49d2be05d972c5f6050dbd307e51df2e68e8e0c83d8ee15b9af34b86e83756f" Oct 30 22:45:15.536966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:15.536773 5139 scope.go:122] "RemoveContainer" containerID="4b08448519186ea76b66ebd7dbe62c09026f1d9c474d2740200712abf00c763a" Oct 30 22:45:15.940312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:15.940283 5139 generic.go:356] "Generic (PLEG): container finished" podID="ef28c061-9ced-41a5-9764-e087e66668a9" containerID="9ca47d3730c3437b31b4d9d1f918c56c883302a1b9af06ed7d123eda067b6200" exitCode=60 Oct 30 22:45:15.940499 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:15.940349 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" event={"ID":"ef28c061-9ced-41a5-9764-e087e66668a9","Type":"ContainerDied","Data":"9ca47d3730c3437b31b4d9d1f918c56c883302a1b9af06ed7d123eda067b6200"} Oct 30 22:45:15.940499 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:15.940423 5139 scope.go:122] "RemoveContainer" containerID="ce939d9d89231dc4f802990d5a43521425d734ff18a70444e0932b9caee1f82d" Oct 30 22:45:15.941172 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:15.940809 5139 scope.go:122] "RemoveContainer" containerID="9ca47d3730c3437b31b4d9d1f918c56c883302a1b9af06ed7d123eda067b6200" Oct 30 22:45:15.941172 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:15.941128 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg_odh-ai-gateway-infra(ef28c061-9ced-41a5-9764-e087e66668a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" Oct 30 22:45:15.942203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:15.942182 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" event={"ID":"9d469dac-2a7b-4e33-bd97-9a17b664e1a1","Type":"ContainerStarted","Data":"63caa621e9f7bd845799dbe72177d405f02ea022a030221edfb0fcf997ac4494"} Oct 30 22:45:16.951550 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:16.951515 5139 generic.go:356] "Generic (PLEG): container finished" podID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerID="63caa621e9f7bd845799dbe72177d405f02ea022a030221edfb0fcf997ac4494" exitCode=60 Oct 30 22:45:16.952059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:16.951588 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" event={"ID":"9d469dac-2a7b-4e33-bd97-9a17b664e1a1","Type":"ContainerDied","Data":"63caa621e9f7bd845799dbe72177d405f02ea022a030221edfb0fcf997ac4494"} Oct 30 22:45:16.952059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:16.951666 5139 scope.go:122] "RemoveContainer" containerID="4b08448519186ea76b66ebd7dbe62c09026f1d9c474d2740200712abf00c763a" Oct 30 22:45:16.952059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:16.952026 5139 scope.go:122] "RemoveContainer" containerID="63caa621e9f7bd845799dbe72177d405f02ea022a030221edfb0fcf997ac4494" Oct 30 22:45:16.952392 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:16.952369 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf_odh-ai-gateway-infra(9d469dac-2a7b-4e33-bd97-9a17b664e1a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" Oct 30 22:45:16.953598 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:16.953577 5139 generic.go:356] "Generic (PLEG): container finished" podID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerID="6d1bd552f3e432004394f7766c2c2c9f5f488510bf124e392aaf798d8310d6b4" exitCode=60 Oct 30 22:45:16.953729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:16.953613 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" event={"ID":"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c","Type":"ContainerDied","Data":"6d1bd552f3e432004394f7766c2c2c9f5f488510bf124e392aaf798d8310d6b4"} Oct 30 22:45:16.953915 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:16.953902 5139 scope.go:122] "RemoveContainer" containerID="6d1bd552f3e432004394f7766c2c2c9f5f488510bf124e392aaf798d8310d6b4" Oct 30 22:45:16.954138 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:16.954122 5139 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6_odh-ai-gateway-infra(cfb611b7-5ee2-4456-9520-e8fbc2f5f80c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" Oct 30 22:45:16.963406 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:16.963389 5139 scope.go:122] "RemoveContainer" containerID="c49d2be05d972c5f6050dbd307e51df2e68e8e0c83d8ee15b9af34b86e83756f" Oct 30 22:45:17.061219 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.061175 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg"] Oct 30 22:45:17.168785 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.168745 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf"] Oct 30 22:45:17.211046 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.210982 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:17.340372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.340330 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef28c061-9ced-41a5-9764-e087e66668a9-kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/ef28c061-9ced-41a5-9764-e087e66668a9-kube-api-access-dq7b4\") pod \"ef28c061-9ced-41a5-9764-e087e66668a9\" (UID: \"ef28c061-9ced-41a5-9764-e087e66668a9\") " Oct 30 22:45:17.340639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.340439 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef28c061-9ced-41a5-9764-e087e66668a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c061-9ced-41a5-9764-e087e66668a9-maas-api-service-ca\") pod \"ef28c061-9ced-41a5-9764-e087e66668a9\" (UID: \"ef28c061-9ced-41a5-9764-e087e66668a9\") " Oct 30 22:45:17.340830 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.340805 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef28c061-9ced-41a5-9764-e087e66668a9-maas-api-service-ca" pod "ef28c061-9ced-41a5-9764-e087e66668a9" (UID: "ef28c061-9ced-41a5-9764-e087e66668a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:17.341028 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.341010 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef28c061-9ced-41a5-9764-e087e66668a9-maas-api-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.342497 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.342477 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef28c061-9ced-41a5-9764-e087e66668a9-kube-api-access-dq7b4" pod "ef28c061-9ced-41a5-9764-e087e66668a9" (UID: "ef28c061-9ced-41a5-9764-e087e66668a9"). InnerVolumeSpecName "kube-api-access-dq7b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:17.442437 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.442378 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dq7b4\" (UniqueName: \"kubernetes.io/projected/ef28c061-9ced-41a5-9764-e087e66668a9-kube-api-access-dq7b4\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:17.964534 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.964510 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" Oct 30 22:45:17.964934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.964501 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg" event={"ID":"ef28c061-9ced-41a5-9764-e087e66668a9","Type":"ContainerDied","Data":"2f021dd53434dce047e0dbbfe0af84e7825a9e188253b5a92295c18ca3b19f74"} Oct 30 22:45:17.964934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:17.964666 5139 scope.go:122] "RemoveContainer" containerID="9ca47d3730c3437b31b4d9d1f918c56c883302a1b9af06ed7d123eda067b6200" Oct 30 22:45:18.128060 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.128036 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:18.175562 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.175535 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg"] Oct 30 22:45:18.180215 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.180185 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29890005-mlwvg"] Oct 30 22:45:18.182994 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.182968 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6"] Oct 30 22:45:18.254059 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.254030 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-maas-api-service-ca\") pod \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\" (UID: \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\") " Oct 30 22:45:18.254289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.254254 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-kube-api-access-cdp6g\" (UniqueName: \"kubernetes.io/projected/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-kube-api-access-cdp6g\") pod \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\" (UID: \"9d469dac-2a7b-4e33-bd97-9a17b664e1a1\") " Oct 30 22:45:18.254504 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.254478 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-maas-api-service-ca" pod "9d469dac-2a7b-4e33-bd97-9a17b664e1a1" (UID: "9d469dac-2a7b-4e33-bd97-9a17b664e1a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:18.254800 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.254781 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-maas-api-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.257149 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.257117 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-kube-api-access-cdp6g" pod "9d469dac-2a7b-4e33-bd97-9a17b664e1a1" (UID: "9d469dac-2a7b-4e33-bd97-9a17b664e1a1"). InnerVolumeSpecName "kube-api-access-cdp6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:18.328127 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.328099 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:18.356030 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.356006 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdp6g\" (UniqueName: \"kubernetes.io/projected/9d469dac-2a7b-4e33-bd97-9a17b664e1a1-kube-api-access-cdp6g\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.457195 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.457168 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-maas-api-service-ca\") pod \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\" (UID: \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\") " Oct 30 22:45:18.457335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.457313 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-kube-api-access-xhrx5\" (UniqueName: \"kubernetes.io/projected/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-kube-api-access-xhrx5\") pod \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\" (UID: \"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c\") " Oct 30 22:45:18.457548 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.457527 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-maas-api-service-ca" pod "cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" (UID: "cfb611b7-5ee2-4456-9520-e8fbc2f5f80c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:18.457693 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.457677 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-maas-api-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.459368 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.459345 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-kube-api-access-xhrx5" pod "cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" (UID: "cfb611b7-5ee2-4456-9520-e8fbc2f5f80c"). InnerVolumeSpecName "kube-api-access-xhrx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:18.559170 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.559132 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhrx5\" (UniqueName: \"kubernetes.io/projected/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c-kube-api-access-xhrx5\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:18.972793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.972714 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" Oct 30 22:45:18.973168 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.972711 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf" event={"ID":"9d469dac-2a7b-4e33-bd97-9a17b664e1a1","Type":"ContainerDied","Data":"40e5ce63ab219d7093d2643f74c0675446ff5b6227965530daa8e32116a4b57e"} Oct 30 22:45:18.973168 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.972879 5139 scope.go:122] "RemoveContainer" containerID="63caa621e9f7bd845799dbe72177d405f02ea022a030221edfb0fcf997ac4494" Oct 30 22:45:18.974239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.974220 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" Oct 30 22:45:18.974344 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.974269 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6" event={"ID":"cfb611b7-5ee2-4456-9520-e8fbc2f5f80c","Type":"ContainerDied","Data":"20362f5a48ab906ddd0d71716607a8e20bb4dabd56dba3097dc9d85213a8c13e"} Oct 30 22:45:18.984069 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:18.984048 5139 scope.go:122] "RemoveContainer" containerID="6d1bd552f3e432004394f7766c2c2c9f5f488510bf124e392aaf798d8310d6b4" Oct 30 22:45:19.130448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:19.130398 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6"] Oct 30 22:45:19.134671 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:19.134644 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-2cc983-29890005-nf2n6"] Oct 30 22:45:19.236837 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:19.236747 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf"] Oct 30 22:45:19.240728 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:19.240697 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-03e614-29890005-kvhqf"] Oct 30 22:45:19.544727 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:19.544698 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" path="/var/lib/kubelet/pods/9d469dac-2a7b-4e33-bd97-9a17b664e1a1/volumes" Oct 30 22:45:19.545158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:19.545143 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" path="/var/lib/kubelet/pods/cfb611b7-5ee2-4456-9520-e8fbc2f5f80c/volumes" Oct 30 22:45:19.545533 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:19.545520 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" path="/var/lib/kubelet/pods/ef28c061-9ced-41a5-9764-e087e66668a9/volumes" Oct 30 22:45:19.805204 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:19.805127 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:45:21.863729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.863691 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56"] Oct 30 22:45:21.864233 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864218 5139 cpu_manager.go:414] "containerMap: removing container" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:21.864296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864233 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:21.864296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864250 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:21.864296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864255 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:21.864296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864269 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:21.864296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864274 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:21.864296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864290 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:21.864296 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864294 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864318 5139 cpu_manager.go:414] "containerMap: removing container" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864323 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864330 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864335 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864451 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864460 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864467 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864473 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864480 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:21.864670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.864487 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:21.868979 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.868957 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.896905 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.896882 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-metrics-certs\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.897050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.897034 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn98j\" (UniqueName: \"kubernetes.io/projected/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-kube-api-access-gn98j\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.897152 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.897130 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.897338 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.897319 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.917926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.917899 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-7a134d-serving-cert\"" Oct 30 22:45:21.967238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.967211 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56"] Oct 30 22:45:21.998487 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.998452 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn98j\" (UniqueName: \"kubernetes.io/projected/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-kube-api-access-gn98j\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.998656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.998567 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.998656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.998607 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.998780 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:21.998683 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-serving-cert: secret "maas-api-e2e-extmod-b-7a134d-serving-cert" not found Oct 30 22:45:21.998780 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:21.998765 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls podName:09bccc97-05ea-4d94-8887-c0d6f1e3bd19 nodeName:}" failed. No retries permitted until 2026-10-30 22:45:22.498741177 +0000 UTC m=+4031.090026664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls") pod "maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" (UID: "09bccc97-05ea-4d94-8887-c0d6f1e3bd19") : secret "maas-api-e2e-extmod-b-7a134d-serving-cert" not found Oct 30 22:45:21.999732 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.998690 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-metrics-certs\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:21.999732 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:21.999395 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:22.010383 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:22.001495 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-metrics-certs\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:22.066377 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:22.066343 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn98j\" (UniqueName: \"kubernetes.io/projected/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-kube-api-access-gn98j\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:22.506388 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:22.506355 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:22.508764 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:22.508740 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls\") pod \"maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:22.781105 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:22.781024 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:23.154692 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:23.154602 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56"] Oct 30 22:45:23.161523 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:45:23.161489 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bccc97_05ea_4d94_8887_c0d6f1e3bd19.slice/crio-cc3715b1496c0639363dcf45caf9e2edac5a8021de27b938a1619d143eb58946 WatchSource:0}: Error finding container cc3715b1496c0639363dcf45caf9e2edac5a8021de27b938a1619d143eb58946: Status 404 returned error can't find the container with id cc3715b1496c0639363dcf45caf9e2edac5a8021de27b938a1619d143eb58946 Oct 30 22:45:24.009484 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:24.009440 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" event={"ID":"09bccc97-05ea-4d94-8887-c0d6f1e3bd19","Type":"ContainerStarted","Data":"cc3715b1496c0639363dcf45caf9e2edac5a8021de27b938a1619d143eb58946"} Oct 30 22:45:25.017005 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:25.016960 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" event={"ID":"09bccc97-05ea-4d94-8887-c0d6f1e3bd19","Type":"ContainerStarted","Data":"958dba0cfaaeba443a3420772f1c79fd738e523afd1ec350e84ac0be8705e72b"} Oct 30 22:45:25.017406 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:25.017041 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:25.128215 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:25.128155 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" podStartSLOduration=3.143387176 podStartE2EDuration="4.128142251s" totalImagesPullingTime="984.755075ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:25.127698759 +0000 UTC m=+4033.718984245" watchObservedRunningTime="2026-10-30 22:45:25.128142251 +0000 UTC m=+4033.719427737" Oct 30 22:45:31.028177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:31.028144 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:45:33.731767 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:33.731723 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps"] Oct 30 22:45:33.732231 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:33.732043 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" containerID="cri-o://42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554" gracePeriod=30 Oct 30 22:45:38.903659 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:38.903619 5139 patch_prober.go:29] interesting pod/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" start-of-body= Oct 30 22:45:38.904107 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:38.903687 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.117:15021/healthz/ready\": dial tcp 10.134.0.117:15021: connect: connection refused" Oct 30 22:45:38.996951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:38.996881 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:45:39.100665 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.100631 5139 generic.go:356] "Generic (PLEG): container finished" podID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerID="42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554" exitCode=0 Oct 30 22:45:39.100873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.100698 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" Oct 30 22:45:39.100873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.100709 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" event={"ID":"8da5aadf-467a-4499-ae82-f7e05e2bde19","Type":"ContainerDied","Data":"42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554"} Oct 30 22:45:39.100873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.100762 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps" event={"ID":"8da5aadf-467a-4499-ae82-f7e05e2bde19","Type":"ContainerDied","Data":"e7c79f4fca5d397289eb4366db539e5ecb9a0688a1971c80bf8de3f021be182a"} Oct 30 22:45:39.100873 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.100795 5139 scope.go:122] "RemoveContainer" containerID="42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554" Oct 30 22:45:39.110267 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.110243 5139 scope.go:122] "RemoveContainer" containerID="7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d" Oct 30 22:45:39.118891 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.118871 5139 scope.go:122] "RemoveContainer" containerID="42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554" Oct 30 22:45:39.119191 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:39.119167 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554\": container with ID starting with 42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554 not found: ID does not exist" containerID="42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554" Oct 30 22:45:39.119298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.119197 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554"} err="failed to get container status \"42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554\": rpc error: code = NotFound desc = could not find container \"42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554\": container with ID starting with 42cfdba36ea7c7be6ee554acb8661ffa3eb12a367b602562cdacf34295c94554 not found: ID does not exist" Oct 30 22:45:39.119298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.119216 5139 scope.go:122] "RemoveContainer" containerID="7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d" Oct 30 22:45:39.119493 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:39.119476 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d\": container with ID starting with 7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d not found: ID does not exist" containerID="7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d" Oct 30 22:45:39.119567 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.119497 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d"} err="failed to get container status \"7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d\": rpc error: code = NotFound desc = could not find container \"7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d\": container with ID starting with 7b63f352dfa82adc82eb95744a13d12209333d0e80c21057ae61010d1243594d not found: ID does not exist" Oct 30 22:45:39.120852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.120831 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-credential-socket\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.120956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.120888 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-token\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-token\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.120956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.120918 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-data\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.120956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.120954 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-kube-api-access-nrkn7\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-kube-api-access-nrkn7\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.121131 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121106 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8da5aadf-467a-4499-ae82-f7e05e2bde19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da5aadf-467a-4499-ae82-f7e05e2bde19-istiod-ca-cert\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.121190 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121143 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-certs\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.121190 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121144 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-credential-socket" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:39.121190 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121170 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-podinfo\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.121605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121192 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-data" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:39.121605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121223 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-envoy\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.121605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121288 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-socket\") pod \"8da5aadf-467a-4499-ae82-f7e05e2bde19\" (UID: \"8da5aadf-467a-4499-ae82-f7e05e2bde19\") " Oct 30 22:45:39.121605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121375 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-certs" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:39.122231 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121680 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-socket" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:39.122231 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.121547 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da5aadf-467a-4499-ae82-f7e05e2bde19-istiod-ca-cert" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:45:39.122539 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.122520 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.122676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.122662 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.122758 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.122748 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.122840 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.122821 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8da5aadf-467a-4499-ae82-f7e05e2bde19-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.122940 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.122842 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.124180 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.124153 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-podinfo" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:45:39.124285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.124210 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-kube-api-access-nrkn7" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "kube-api-access-nrkn7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:39.124587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.124561 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-envoy" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:45:39.125033 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.124998 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-token" pod "8da5aadf-467a-4499-ae82-f7e05e2bde19" (UID: "8da5aadf-467a-4499-ae82-f7e05e2bde19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:45:39.223927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.223891 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.223927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.223922 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.223927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.223932 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrkn7\" (UniqueName: \"kubernetes.io/projected/8da5aadf-467a-4499-ae82-f7e05e2bde19-kube-api-access-nrkn7\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.224163 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.223941 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8da5aadf-467a-4499-ae82-f7e05e2bde19-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:45:39.518949 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.518915 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps"] Oct 30 22:45:39.524043 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.524017 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-5bb0b1-openshift-default-7b9f69bf65-wvdps"] Oct 30 22:45:39.543264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:39.543233 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" path="/var/lib/kubelet/pods/8da5aadf-467a-4499-ae82-f7e05e2bde19/volumes" Oct 30 22:45:49.805729 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:49.805686 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:45:54.302035 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.301950 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x"] Oct 30 22:45:54.302529 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302513 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:54.302580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302530 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:54.302580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302542 5139 cpu_manager.go:414] "containerMap: removing container" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:54.302580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302547 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:54.302580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302559 5139 cpu_manager.go:414] "containerMap: removing container" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" Oct 30 22:45:54.302580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302564 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" Oct 30 22:45:54.302580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302576 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:54.302580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302581 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:54.302796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302681 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfb611b7-5ee2-4456-9520-e8fbc2f5f80c" containerName="cleanup" Oct 30 22:45:54.302796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302689 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" Oct 30 22:45:54.302796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302697 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d469dac-2a7b-4e33-bd97-9a17b664e1a1" containerName="cleanup" Oct 30 22:45:54.302796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.302705 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef28c061-9ced-41a5-9764-e087e66668a9" containerName="cleanup" Oct 30 22:45:54.306278 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.306258 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.355347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.355319 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-6e0b99-serving-cert\"" Oct 30 22:45:54.406517 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.406478 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x"] Oct 30 22:45:54.412354 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.412326 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-metrics-certs\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.412543 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.412495 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjfn\" (UniqueName: \"kubernetes.io/projected/653cda71-29c3-4cd6-b563-71df8adc1c64-kube-api-access-srjfn\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.412621 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.412565 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.412684 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.412625 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653cda71-29c3-4cd6-b563-71df8adc1c64-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.513397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.513355 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-metrics-certs\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.513608 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.513538 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjfn\" (UniqueName: \"kubernetes.io/projected/653cda71-29c3-4cd6-b563-71df8adc1c64-kube-api-access-srjfn\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.513608 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.513577 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.513740 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.513690 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653cda71-29c3-4cd6-b563-71df8adc1c64-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.513798 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:54.513760 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-serving-cert: secret "maas-api-e2e-extmod-a-6e0b99-serving-cert" not found Oct 30 22:45:54.513886 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:45:54.513862 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls podName:653cda71-29c3-4cd6-b563-71df8adc1c64 nodeName:}" failed. No retries permitted until 2026-10-30 22:45:55.013841301 +0000 UTC m=+4063.605126781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls") pod "maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" (UID: "653cda71-29c3-4cd6-b563-71df8adc1c64") : secret "maas-api-e2e-extmod-a-6e0b99-serving-cert" not found Oct 30 22:45:54.514561 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.514540 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653cda71-29c3-4cd6-b563-71df8adc1c64-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.516039 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.516017 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-metrics-certs\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:54.594543 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:54.594178 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjfn\" (UniqueName: \"kubernetes.io/projected/653cda71-29c3-4cd6-b563-71df8adc1c64-kube-api-access-srjfn\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:55.020205 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:55.020171 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:55.022693 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:55.022670 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls\") pod \"maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:55.219035 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:55.218999 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:55.618065 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:55.617722 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x"] Oct 30 22:45:55.622569 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:45:55.622534 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653cda71_29c3_4cd6_b563_71df8adc1c64.slice/crio-07ab89973bfc01e1aa5d9888d64160e11eb87b0c25f69aaf419f1f5034e9a823 WatchSource:0}: Error finding container 07ab89973bfc01e1aa5d9888d64160e11eb87b0c25f69aaf419f1f5034e9a823: Status 404 returned error can't find the container with id 07ab89973bfc01e1aa5d9888d64160e11eb87b0c25f69aaf419f1f5034e9a823 Oct 30 22:45:56.211716 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:56.211679 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" event={"ID":"653cda71-29c3-4cd6-b563-71df8adc1c64","Type":"ContainerStarted","Data":"07ab89973bfc01e1aa5d9888d64160e11eb87b0c25f69aaf419f1f5034e9a823"} Oct 30 22:45:57.218897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:57.218864 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" event={"ID":"653cda71-29c3-4cd6-b563-71df8adc1c64","Type":"ContainerStarted","Data":"b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9"} Oct 30 22:45:57.219320 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:57.219001 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:45:57.354508 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:45:57.354448 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" podStartSLOduration=2.393830714 podStartE2EDuration="3.354397044s" totalImagesPullingTime="960.56633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:57.351290353 +0000 UTC m=+4065.942575834" watchObservedRunningTime="2026-10-30 22:45:57.354397044 +0000 UTC m=+4065.945682530" Oct 30 22:46:02.254180 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:02.254146 5139 generic.go:356] "Generic (PLEG): container finished" podID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerID="0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f" exitCode=0 Oct 30 22:46:02.254646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:02.254222 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" event={"ID":"51e8093e-00ea-4e68-8923-d3556d1ae1e3","Type":"ContainerDied","Data":"0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f"} Oct 30 22:46:02.254736 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:02.254701 5139 scope.go:122] "RemoveContainer" containerID="0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f" Oct 30 22:46:02.256390 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:02.256357 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:46:02.256523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:02.256458 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:46:02.256523 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:02.256494 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:46:03.231958 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:03.231931 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:46:03.261860 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:03.261829 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" event={"ID":"51e8093e-00ea-4e68-8923-d3556d1ae1e3","Type":"ContainerStarted","Data":"708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707"} Oct 30 22:46:03.331674 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:03.331640 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:46:03.650393 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:03.650356 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:46:03.651656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:03.651630 5139 patch_prober.go:29] interesting pod/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:46:03.651794 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:03.651700 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:46:04.650157 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:04.650123 5139 patch_prober.go:29] interesting pod/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:46:04.650638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:04.650178 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:46:05.650225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:05.650194 5139 patch_prober.go:29] interesting pod/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:46:05.650660 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:05.650243 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:46:06.650082 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:06.650049 5139 patch_prober.go:29] interesting pod/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:46:06.650274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:06.650105 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:46:07.650588 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:07.650549 5139 patch_prober.go:29] interesting pod/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 30 22:46:07.650975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:07.650619 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 30 22:46:07.994926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:07.994892 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn"] Oct 30 22:46:07.995116 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:07.995095 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" podUID="d7aedde2-8cea-4844-a490-01605c800cbe" containerName="maas-api" containerID="cri-o://299ec7e776d7d3ba9a5548c8d7bad5c655cbb34891a3ccb83e3bc92763cc3c07" gracePeriod=30 Oct 30 22:46:08.328525 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.328448 5139 generic.go:356] "Generic (PLEG): container finished" podID="d7aedde2-8cea-4844-a490-01605c800cbe" containerID="299ec7e776d7d3ba9a5548c8d7bad5c655cbb34891a3ccb83e3bc92763cc3c07" exitCode=0 Oct 30 22:46:08.328660 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.328523 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" event={"ID":"d7aedde2-8cea-4844-a490-01605c800cbe","Type":"ContainerDied","Data":"299ec7e776d7d3ba9a5548c8d7bad5c655cbb34891a3ccb83e3bc92763cc3c07"} Oct 30 22:46:08.346760 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.346734 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:46:08.521506 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.521455 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls\") pod \"d7aedde2-8cea-4844-a490-01605c800cbe\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " Oct 30 22:46:08.521708 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.521584 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7aedde2-8cea-4844-a490-01605c800cbe-kube-api-access-sc4g9\" (UniqueName: \"kubernetes.io/projected/d7aedde2-8cea-4844-a490-01605c800cbe-kube-api-access-sc4g9\") pod \"d7aedde2-8cea-4844-a490-01605c800cbe\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " Oct 30 22:46:08.521793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.521757 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7aedde2-8cea-4844-a490-01605c800cbe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aedde2-8cea-4844-a490-01605c800cbe-openshift-service-ca\") pod \"d7aedde2-8cea-4844-a490-01605c800cbe\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " Oct 30 22:46:08.521854 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.521827 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-metrics-certs\") pod \"d7aedde2-8cea-4844-a490-01605c800cbe\" (UID: \"d7aedde2-8cea-4844-a490-01605c800cbe\") " Oct 30 22:46:08.522632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.522598 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7aedde2-8cea-4844-a490-01605c800cbe-openshift-service-ca" pod "d7aedde2-8cea-4844-a490-01605c800cbe" (UID: "d7aedde2-8cea-4844-a490-01605c800cbe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:46:08.523983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.523960 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls" pod "d7aedde2-8cea-4844-a490-01605c800cbe" (UID: "d7aedde2-8cea-4844-a490-01605c800cbe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:08.524325 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.524297 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7aedde2-8cea-4844-a490-01605c800cbe-kube-api-access-sc4g9" pod "d7aedde2-8cea-4844-a490-01605c800cbe" (UID: "d7aedde2-8cea-4844-a490-01605c800cbe"). InnerVolumeSpecName "kube-api-access-sc4g9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:46:08.524446 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.524336 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-metrics-certs" pod "d7aedde2-8cea-4844-a490-01605c800cbe" (UID: "d7aedde2-8cea-4844-a490-01605c800cbe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:46:08.628135 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.625538 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7aedde2-8cea-4844-a490-01605c800cbe-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:08.628135 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.625573 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:08.628135 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.625590 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7aedde2-8cea-4844-a490-01605c800cbe-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:08.628135 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.625603 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc4g9\" (UniqueName: \"kubernetes.io/projected/d7aedde2-8cea-4844-a490-01605c800cbe-kube-api-access-sc4g9\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:46:08.654663 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.654636 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:46:08.655640 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:08.655620 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:46:09.336438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:09.336381 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" Oct 30 22:46:09.336657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:09.336453 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn" event={"ID":"d7aedde2-8cea-4844-a490-01605c800cbe","Type":"ContainerDied","Data":"13aee7d3cb57e6c1ad33bd7042c109a953173dbd8aced1a2b0b5ee0d57d5076f"} Oct 30 22:46:09.336657 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:09.336511 5139 scope.go:122] "RemoveContainer" containerID="299ec7e776d7d3ba9a5548c8d7bad5c655cbb34891a3ccb83e3bc92763cc3c07" Oct 30 22:46:09.491071 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:09.491035 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn"] Oct 30 22:46:09.497398 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:09.497369 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-be6ecc-79c4bd6df4-5f2fn"] Oct 30 22:46:09.553630 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:09.553591 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7aedde2-8cea-4844-a490-01605c800cbe" path="/var/lib/kubelet/pods/d7aedde2-8cea-4844-a490-01605c800cbe/volumes" Oct 30 22:46:14.505718 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.505682 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25"] Oct 30 22:46:14.506299 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.506283 5139 cpu_manager.go:414] "containerMap: removing container" podUID="d7aedde2-8cea-4844-a490-01605c800cbe" containerName="maas-api" Oct 30 22:46:14.506343 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.506300 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7aedde2-8cea-4844-a490-01605c800cbe" containerName="maas-api" Oct 30 22:46:14.506343 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.506310 5139 cpu_manager.go:414] "containerMap: removing container" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" Oct 30 22:46:14.506343 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.506316 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" Oct 30 22:46:14.506460 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.506410 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7aedde2-8cea-4844-a490-01605c800cbe" containerName="maas-api" Oct 30 22:46:14.506460 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.506438 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="8da5aadf-467a-4499-ae82-f7e05e2bde19" containerName="istio-proxy" Oct 30 22:46:14.511932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.511909 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.560656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.560627 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-b-6b46c3-serving-cert\"" Oct 30 22:46:14.600966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.600926 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.601168 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.601090 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/17c8ec1b-452e-49b5-8708-4093b854ef9b-kube-api-access-qdfxh\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.601242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.601165 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-metrics-certs\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.601365 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.601339 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8ec1b-452e-49b5-8708-4093b854ef9b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.609852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.609820 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25"] Oct 30 22:46:14.702747 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.702713 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-metrics-certs\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.702926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.702825 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8ec1b-452e-49b5-8708-4093b854ef9b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.702975 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.702938 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.703038 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.703021 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/17c8ec1b-452e-49b5-8708-4093b854ef9b-kube-api-access-qdfxh\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.703119 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:46:14.703098 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-serving-cert: secret "maas-api-e2e-extmod-b-6b46c3-serving-cert" not found Oct 30 22:46:14.703198 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:46:14.703184 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls podName:17c8ec1b-452e-49b5-8708-4093b854ef9b nodeName:}" failed. No retries permitted until 2026-10-30 22:46:15.203162254 +0000 UTC m=+4083.794447737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls") pod "maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" (UID: "17c8ec1b-452e-49b5-8708-4093b854ef9b") : secret "maas-api-e2e-extmod-b-6b46c3-serving-cert" not found Oct 30 22:46:14.703628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.703607 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8ec1b-452e-49b5-8708-4093b854ef9b-openshift-service-ca\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.705335 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.705310 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-metrics-certs\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:14.764853 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:14.764761 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/17c8ec1b-452e-49b5-8708-4093b854ef9b-kube-api-access-qdfxh\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:15.209881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:15.209796 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:15.212209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:15.212183 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls\") pod \"maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:15.425034 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:15.424999 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:15.814334 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:15.814255 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25"] Oct 30 22:46:15.820742 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:46:15.820714 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c8ec1b_452e_49b5_8708_4093b854ef9b.slice/crio-85d07f7188fc488432a07ddbb140e1ea9c4e5b5f3207fcb1d221e89f83431086 WatchSource:0}: Error finding container 85d07f7188fc488432a07ddbb140e1ea9c4e5b5f3207fcb1d221e89f83431086: Status 404 returned error can't find the container with id 85d07f7188fc488432a07ddbb140e1ea9c4e5b5f3207fcb1d221e89f83431086 Oct 30 22:46:16.379529 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:16.379488 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" event={"ID":"17c8ec1b-452e-49b5-8708-4093b854ef9b","Type":"ContainerStarted","Data":"85d07f7188fc488432a07ddbb140e1ea9c4e5b5f3207fcb1d221e89f83431086"} Oct 30 22:46:17.393336 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:17.393297 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" event={"ID":"17c8ec1b-452e-49b5-8708-4093b854ef9b","Type":"ContainerStarted","Data":"b34c3490a15bf4c90454401574745187b48df949e1807a18d25471a64ccc0567"} Oct 30 22:46:17.393760 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:17.393384 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:17.505699 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:17.505641 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" podStartSLOduration=2.636633091 podStartE2EDuration="3.505625297s" totalImagesPullingTime="868.992206ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:17.504125095 +0000 UTC m=+4086.095410584" watchObservedRunningTime="2026-10-30 22:46:17.505625297 +0000 UTC m=+4086.096910784" Oct 30 22:46:19.814281 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:46:19.814236 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:46:23.405979 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:23.405952 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:46:26.466732 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:26.466697 5139 generic.go:356] "Generic (PLEG): container finished" podID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerID="d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5" exitCode=0 Oct 30 22:46:26.467168 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:26.466771 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" event={"ID":"3c4f44d4-a36e-4779-b7ab-908329d4691b","Type":"ContainerDied","Data":"d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5"} Oct 30 22:46:26.467227 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:26.467206 5139 scope.go:122] "RemoveContainer" containerID="d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5" Oct 30 22:46:26.469098 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:26.469069 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:46:26.469179 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:26.469138 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:46:26.469179 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:26.469167 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:46:27.487583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:27.487542 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:46:27.492983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:27.492955 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" event={"ID":"3c4f44d4-a36e-4779-b7ab-908329d4691b","Type":"ContainerStarted","Data":"426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584"} Oct 30 22:46:28.346656 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:28.346620 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:46:28.347981 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:28.347959 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:46:28.348076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:28.348054 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 22:46:29.346960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:29.346924 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:46:29.347363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:29.346979 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 22:46:30.346878 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:30.346834 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:46:30.347077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:30.346895 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 22:46:31.346509 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:31.346470 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:46:31.346700 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:31.346528 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 22:46:32.346639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:32.346602 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 22:46:32.347051 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:32.346661 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 22:46:33.352739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:33.352709 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:46:33.353185 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:33.352996 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:46:33.353946 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:33.353921 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:46:49.805212 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:46:49.805177 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:46:52.440786 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:46:52.440736 5139 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal" podUID="3ab1a8fb7b394ed5c2ad3173d5294ab9" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:47:19.804987 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:47:19.804948 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:47:34.853937 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:34.853902 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x"] Oct 30 22:47:34.854461 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:34.854113 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" podUID="653cda71-29c3-4cd6-b563-71df8adc1c64" containerName="maas-api" containerID="cri-o://b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9" gracePeriod=30 Oct 30 22:47:35.121097 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.121044 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:47:35.166119 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.166091 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls\") pod \"653cda71-29c3-4cd6-b563-71df8adc1c64\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " Oct 30 22:47:35.166230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.166152 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653cda71-29c3-4cd6-b563-71df8adc1c64-kube-api-access-srjfn\" (UniqueName: \"kubernetes.io/projected/653cda71-29c3-4cd6-b563-71df8adc1c64-kube-api-access-srjfn\") pod \"653cda71-29c3-4cd6-b563-71df8adc1c64\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " Oct 30 22:47:35.166230 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.166205 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-metrics-certs\") pod \"653cda71-29c3-4cd6-b563-71df8adc1c64\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " Oct 30 22:47:35.166322 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.166313 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653cda71-29c3-4cd6-b563-71df8adc1c64-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653cda71-29c3-4cd6-b563-71df8adc1c64-openshift-service-ca\") pod \"653cda71-29c3-4cd6-b563-71df8adc1c64\" (UID: \"653cda71-29c3-4cd6-b563-71df8adc1c64\") " Oct 30 22:47:35.166646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.166620 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653cda71-29c3-4cd6-b563-71df8adc1c64-openshift-service-ca" pod "653cda71-29c3-4cd6-b563-71df8adc1c64" (UID: "653cda71-29c3-4cd6-b563-71df8adc1c64"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:35.166892 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.166870 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/653cda71-29c3-4cd6-b563-71df8adc1c64-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:35.168180 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.168154 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls" pod "653cda71-29c3-4cd6-b563-71df8adc1c64" (UID: "653cda71-29c3-4cd6-b563-71df8adc1c64"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:35.168284 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.168241 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-metrics-certs" pod "653cda71-29c3-4cd6-b563-71df8adc1c64" (UID: "653cda71-29c3-4cd6-b563-71df8adc1c64"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:47:35.168333 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.168297 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653cda71-29c3-4cd6-b563-71df8adc1c64-kube-api-access-srjfn" pod "653cda71-29c3-4cd6-b563-71df8adc1c64" (UID: "653cda71-29c3-4cd6-b563-71df8adc1c64"). InnerVolumeSpecName "kube-api-access-srjfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:35.268271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.268232 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srjfn\" (UniqueName: \"kubernetes.io/projected/653cda71-29c3-4cd6-b563-71df8adc1c64-kube-api-access-srjfn\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:35.268271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.268265 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:35.268492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.268279 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/653cda71-29c3-4cd6-b563-71df8adc1c64-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:35.952909 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.952817 5139 generic.go:356] "Generic (PLEG): container finished" podID="653cda71-29c3-4cd6-b563-71df8adc1c64" containerID="b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9" exitCode=0 Oct 30 22:47:35.952909 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.952866 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" Oct 30 22:47:35.952909 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.952878 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" event={"ID":"653cda71-29c3-4cd6-b563-71df8adc1c64","Type":"ContainerDied","Data":"b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9"} Oct 30 22:47:35.953503 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.952924 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x" event={"ID":"653cda71-29c3-4cd6-b563-71df8adc1c64","Type":"ContainerDied","Data":"07ab89973bfc01e1aa5d9888d64160e11eb87b0c25f69aaf419f1f5034e9a823"} Oct 30 22:47:35.953503 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.952948 5139 scope.go:122] "RemoveContainer" containerID="b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9" Oct 30 22:47:35.962872 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.962842 5139 scope.go:122] "RemoveContainer" containerID="b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9" Oct 30 22:47:35.963165 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:47:35.963146 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9\": container with ID starting with b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9 not found: ID does not exist" containerID="b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9" Oct 30 22:47:35.963232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:35.963175 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9"} err="failed to get container status \"b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9\": rpc error: code = NotFound desc = could not find container \"b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9\": container with ID starting with b381882bf6cdad961b8f94a5c95e30cda2cfbc6132cbf38a4e4a8ac3b9b9c2a9 not found: ID does not exist" Oct 30 22:47:36.107886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:36.107857 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x"] Oct 30 22:47:36.153995 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:36.153965 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-6e0b99-76567f47d9-2m78x"] Oct 30 22:47:37.544145 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:37.544110 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653cda71-29c3-4cd6-b563-71df8adc1c64" path="/var/lib/kubelet/pods/653cda71-29c3-4cd6-b563-71df8adc1c64/volumes" Oct 30 22:47:41.970881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:41.970848 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk"] Oct 30 22:47:41.971274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:41.971086 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" containerID="cri-o://708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707" gracePeriod=30 Oct 30 22:47:43.931912 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:43.931876 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882"] Oct 30 22:47:43.932345 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:43.932108 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" containerID="cri-o://426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584" gracePeriod=30 Oct 30 22:47:47.218325 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.218298 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:47:47.319090 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319063 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istiod-ca-cert\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319244 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319113 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-kube-api-access-sqfht\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-kube-api-access-sqfht\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319244 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319172 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-credential-socket\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319244 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319225 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-data\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319394 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319250 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-podinfo\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319394 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319281 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-certs\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319394 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319361 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-socket\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319500 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-envoy\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319547 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-token\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-token\") pod \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\" (UID: \"51e8093e-00ea-4e68-8923-d3556d1ae1e3\") " Oct 30 22:47:47.319593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319551 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-credential-socket" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.319593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319573 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-data" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.319811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319648 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-certs" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.319811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319655 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istiod-ca-cert" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:47.319811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.319769 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-socket" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.320053 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.320023 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.320053 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.320051 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.320257 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.320062 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.320257 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.320071 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.320257 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.320079 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.321561 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.321531 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-kube-api-access-sqfht" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "kube-api-access-sqfht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:47.321654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.321560 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-envoy" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:47.321654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.321598 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-podinfo" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:47.321760 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.321743 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-token" pod "51e8093e-00ea-4e68-8923-d3556d1ae1e3" (UID: "51e8093e-00ea-4e68-8923-d3556d1ae1e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:47.421847 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.421819 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.421847 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.421848 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqfht\" (UniqueName: \"kubernetes.io/projected/51e8093e-00ea-4e68-8923-d3556d1ae1e3-kube-api-access-sqfht\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.422047 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.421863 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:47.422047 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:47.421876 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51e8093e-00ea-4e68-8923-d3556d1ae1e3-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:48.023210 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.023177 5139 generic.go:356] "Generic (PLEG): container finished" podID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerID="708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707" exitCode=0 Oct 30 22:47:48.023435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.023223 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" Oct 30 22:47:48.023435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.023263 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" event={"ID":"51e8093e-00ea-4e68-8923-d3556d1ae1e3","Type":"ContainerDied","Data":"708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707"} Oct 30 22:47:48.023435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.023330 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk" event={"ID":"51e8093e-00ea-4e68-8923-d3556d1ae1e3","Type":"ContainerDied","Data":"6ae7608276d9528e4f35980f6821244865b275d42c2d70d94a36c678509583d3"} Oct 30 22:47:48.023435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.023364 5139 scope.go:122] "RemoveContainer" containerID="708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707" Oct 30 22:47:48.033560 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.033540 5139 scope.go:122] "RemoveContainer" containerID="0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f" Oct 30 22:47:48.043440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.043407 5139 scope.go:122] "RemoveContainer" containerID="708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707" Oct 30 22:47:48.043689 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:47:48.043666 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707\": container with ID starting with 708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707 not found: ID does not exist" containerID="708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707" Oct 30 22:47:48.043743 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.043696 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707"} err="failed to get container status \"708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707\": rpc error: code = NotFound desc = could not find container \"708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707\": container with ID starting with 708af4c103b76dcb77944b6e3c73ee998be878a99f92964767671ac3abacb707 not found: ID does not exist" Oct 30 22:47:48.043743 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.043712 5139 scope.go:122] "RemoveContainer" containerID="0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f" Oct 30 22:47:48.043968 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:47:48.043951 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f\": container with ID starting with 0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f not found: ID does not exist" containerID="0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f" Oct 30 22:47:48.044015 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.043973 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f"} err="failed to get container status \"0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f\": rpc error: code = NotFound desc = could not find container \"0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f\": container with ID starting with 0610b40aedf82cd3c7482ed0673a1e51b263aeab8376a72a0b11423eca3a873f not found: ID does not exist" Oct 30 22:47:48.233248 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.233214 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk"] Oct 30 22:47:48.279621 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.279548 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-5a5534-openshift-default-66cbb544f7-xpfbk"] Oct 30 22:47:48.355332 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.355290 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:47:48.355502 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:48.355358 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:47:49.205969 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.205911 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:47:49.345313 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345281 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-data\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.345817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345354 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-token\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-token\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.345817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345385 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-podinfo\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.345817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345440 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-envoy\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.345817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345547 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-certs\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.345817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345637 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-data" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:49.345817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345649 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4f44d4-a36e-4779-b7ab-908329d4691b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4f44d4-a36e-4779-b7ab-908329d4691b-istiod-ca-cert\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.345817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345756 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-credential-socket\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.346205 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345836 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-kube-api-access-dlzww\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-kube-api-access-dlzww\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.346205 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.345909 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-socket\") pod \"3c4f44d4-a36e-4779-b7ab-908329d4691b\" (UID: \"3c4f44d4-a36e-4779-b7ab-908329d4691b\") " Oct 30 22:47:49.346205 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346060 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-certs" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:49.346205 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346069 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-credential-socket" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:49.346205 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346093 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4f44d4-a36e-4779-b7ab-908329d4691b-istiod-ca-cert" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:47:49.346507 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346363 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-socket" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:49.346886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346862 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.346949 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346894 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c4f44d4-a36e-4779-b7ab-908329d4691b-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.346949 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346909 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.346949 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346924 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.346949 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.346939 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.348143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.348103 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-token" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:49.348310 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.348286 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-podinfo" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:47:49.348951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.348919 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-envoy" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:47:49.348951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.348929 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-kube-api-access-dlzww" pod "3c4f44d4-a36e-4779-b7ab-908329d4691b" (UID: "3c4f44d4-a36e-4779-b7ab-908329d4691b"). InnerVolumeSpecName "kube-api-access-dlzww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:47:49.448805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.448770 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlzww\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-kube-api-access-dlzww\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.448805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.448802 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.448805 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.448813 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.449043 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.448823 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c4f44d4-a36e-4779-b7ab-908329d4691b-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:47:49.543886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:49.543848 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" path="/var/lib/kubelet/pods/51e8093e-00ea-4e68-8923-d3556d1ae1e3/volumes" Oct 30 22:47:49.805804 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:47:49.805720 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:47:50.037284 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.037250 5139 generic.go:356] "Generic (PLEG): container finished" podID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerID="426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584" exitCode=0 Oct 30 22:47:50.037507 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.037304 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" Oct 30 22:47:50.037507 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.037339 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" event={"ID":"3c4f44d4-a36e-4779-b7ab-908329d4691b","Type":"ContainerDied","Data":"426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584"} Oct 30 22:47:50.037507 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.037404 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882" event={"ID":"3c4f44d4-a36e-4779-b7ab-908329d4691b","Type":"ContainerDied","Data":"e83d8bcc3daf8f046201f7742f562ec21d10784e50fbb0fc69b89e6827a160dd"} Oct 30 22:47:50.037507 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.037461 5139 scope.go:122] "RemoveContainer" containerID="426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584" Oct 30 22:47:50.047487 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.047470 5139 scope.go:122] "RemoveContainer" containerID="d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5" Oct 30 22:47:50.056877 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.056682 5139 scope.go:122] "RemoveContainer" containerID="426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584" Oct 30 22:47:50.056986 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:47:50.056964 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584\": container with ID starting with 426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584 not found: ID does not exist" containerID="426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584" Oct 30 22:47:50.057054 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.057003 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584"} err="failed to get container status \"426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584\": rpc error: code = NotFound desc = could not find container \"426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584\": container with ID starting with 426d3d7708eb25c0faee764ef62a5fdf366b6bd44587adfd1662f54b767dd584 not found: ID does not exist" Oct 30 22:47:50.057054 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.057029 5139 scope.go:122] "RemoveContainer" containerID="d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5" Oct 30 22:47:50.057331 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:47:50.057311 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5\": container with ID starting with d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5 not found: ID does not exist" containerID="d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5" Oct 30 22:47:50.057384 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.057338 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5"} err="failed to get container status \"d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5\": rpc error: code = NotFound desc = could not find container \"d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5\": container with ID starting with d2ec9d3624d1bbac22a279a6b636d6e1774a94506717645650b2d7238e1cb0a5 not found: ID does not exist" Oct 30 22:47:50.239433 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.239394 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882"] Oct 30 22:47:50.243931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:50.243904 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-8x882"] Oct 30 22:47:51.554817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:51.554781 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" path="/var/lib/kubelet/pods/3c4f44d4-a36e-4779-b7ab-908329d4691b/volumes" Oct 30 22:47:57.892646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:57.892617 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5"] Oct 30 22:47:58.008643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:47:58.008611 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz"] Oct 30 22:48:02.893334 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:02.893254 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" podUID="0828b8c5-62d8-48ac-b9ac-06dd7373aeb8" containerName="payload-processing" containerID="cri-o://70ec919562b3deb403b7eea992e65f2dd36039d5c5501a3a8949c4cfe259df00" gracePeriod=25 Oct 30 22:48:03.009690 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.009630 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" podUID="c5b51f90-7eec-43b1-af98-3a40d6e66104" containerName="payload-pre-processing" containerID="cri-o://56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa" gracePeriod=25 Oct 30 22:48:03.113355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.113317 5139 generic.go:356] "Generic (PLEG): container finished" podID="0828b8c5-62d8-48ac-b9ac-06dd7373aeb8" containerID="70ec919562b3deb403b7eea992e65f2dd36039d5c5501a3a8949c4cfe259df00" exitCode=0 Oct 30 22:48:03.113569 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.113383 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" event={"ID":"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8","Type":"ContainerDied","Data":"70ec919562b3deb403b7eea992e65f2dd36039d5c5501a3a8949c4cfe259df00"} Oct 30 22:48:03.175957 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.175929 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:48:03.216472 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.216435 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-plugins-config-volume\") pod \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\" (UID: \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\") " Oct 30 22:48:03.216670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.216594 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-kube-api-access-xx48d\" (UniqueName: \"kubernetes.io/projected/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-kube-api-access-xx48d\") pod \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\" (UID: \"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8\") " Oct 30 22:48:03.216869 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.216842 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-plugins-config-volume" pod "0828b8c5-62d8-48ac-b9ac-06dd7373aeb8" (UID: "0828b8c5-62d8-48ac-b9ac-06dd7373aeb8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:03.217061 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.217044 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:03.219855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.219819 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-kube-api-access-xx48d" pod "0828b8c5-62d8-48ac-b9ac-06dd7373aeb8" (UID: "0828b8c5-62d8-48ac-b9ac-06dd7373aeb8"). InnerVolumeSpecName "kube-api-access-xx48d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:03.262096 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.262072 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:48:03.318101 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.318067 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5b51f90-7eec-43b1-af98-3a40d6e66104-kube-api-access-vj2gz\" (UniqueName: \"kubernetes.io/projected/c5b51f90-7eec-43b1-af98-3a40d6e66104-kube-api-access-vj2gz\") pod \"c5b51f90-7eec-43b1-af98-3a40d6e66104\" (UID: \"c5b51f90-7eec-43b1-af98-3a40d6e66104\") " Oct 30 22:48:03.318273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.318122 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5b51f90-7eec-43b1-af98-3a40d6e66104-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b51f90-7eec-43b1-af98-3a40d6e66104-plugins-config-volume\") pod \"c5b51f90-7eec-43b1-af98-3a40d6e66104\" (UID: \"c5b51f90-7eec-43b1-af98-3a40d6e66104\") " Oct 30 22:48:03.318397 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.318384 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx48d\" (UniqueName: \"kubernetes.io/projected/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8-kube-api-access-xx48d\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:03.318563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.318532 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5b51f90-7eec-43b1-af98-3a40d6e66104-plugins-config-volume" pod "c5b51f90-7eec-43b1-af98-3a40d6e66104" (UID: "c5b51f90-7eec-43b1-af98-3a40d6e66104"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:03.320305 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.320274 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5b51f90-7eec-43b1-af98-3a40d6e66104-kube-api-access-vj2gz" pod "c5b51f90-7eec-43b1-af98-3a40d6e66104" (UID: "c5b51f90-7eec-43b1-af98-3a40d6e66104"). InnerVolumeSpecName "kube-api-access-vj2gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:03.419899 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.419825 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj2gz\" (UniqueName: \"kubernetes.io/projected/c5b51f90-7eec-43b1-af98-3a40d6e66104-kube-api-access-vj2gz\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:03.419899 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:03.419863 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5b51f90-7eec-43b1-af98-3a40d6e66104-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:04.120862 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.120824 5139 generic.go:356] "Generic (PLEG): container finished" podID="c5b51f90-7eec-43b1-af98-3a40d6e66104" containerID="56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa" exitCode=0 Oct 30 22:48:04.121289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.120877 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" Oct 30 22:48:04.121289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.120901 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" event={"ID":"c5b51f90-7eec-43b1-af98-3a40d6e66104","Type":"ContainerDied","Data":"56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa"} Oct 30 22:48:04.121289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.120953 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz" event={"ID":"c5b51f90-7eec-43b1-af98-3a40d6e66104","Type":"ContainerDied","Data":"c1ec004dfd76e4e6dedfd87680de313aa576bb4b5e80e045b21a9078c5134c49"} Oct 30 22:48:04.121289 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.120973 5139 scope.go:122] "RemoveContainer" containerID="56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa" Oct 30 22:48:04.122532 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.122459 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" Oct 30 22:48:04.122532 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.122467 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5" event={"ID":"0828b8c5-62d8-48ac-b9ac-06dd7373aeb8","Type":"ContainerDied","Data":"39c960906c7bdb96cbc96e2be3b4f28549c70de4371fa71c04d68c41389a8692"} Oct 30 22:48:04.130596 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.130579 5139 scope.go:122] "RemoveContainer" containerID="56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa" Oct 30 22:48:04.130869 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:48:04.130846 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa\": container with ID starting with 56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa not found: ID does not exist" containerID="56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa" Oct 30 22:48:04.130926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.130877 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa"} err="failed to get container status \"56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa\": rpc error: code = NotFound desc = could not find container \"56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa\": container with ID starting with 56add43d51e3a6b09d073ddf2465a5aa75b08cbb92dddd03a11095d2660c3eaa not found: ID does not exist" Oct 30 22:48:04.130926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.130894 5139 scope.go:122] "RemoveContainer" containerID="70ec919562b3deb403b7eea992e65f2dd36039d5c5501a3a8949c4cfe259df00" Oct 30 22:48:04.311864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.311826 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz"] Oct 30 22:48:04.357009 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.356975 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-03e614-86546c967-n96mz"] Oct 30 22:48:04.430879 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.430847 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5"] Oct 30 22:48:04.433900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:04.433868 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-03e614-5f847dd695-kxhq5"] Oct 30 22:48:05.544233 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:05.544197 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0828b8c5-62d8-48ac-b9ac-06dd7373aeb8" path="/var/lib/kubelet/pods/0828b8c5-62d8-48ac-b9ac-06dd7373aeb8/volumes" Oct 30 22:48:05.544793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:05.544647 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5b51f90-7eec-43b1-af98-3a40d6e66104" path="/var/lib/kubelet/pods/c5b51f90-7eec-43b1-af98-3a40d6e66104/volumes" Oct 30 22:48:15.188629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:15.188594 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4"] Oct 30 22:48:19.805200 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:48:19.805163 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:48:26.029903 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.029871 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv"] Oct 30 22:48:26.030404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.030055 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" podUID="5fa931b0-386c-44c1-91bd-b6ec8940ad26" containerName="maas-api" containerID="cri-o://cb322c90ff860d63f7058f9635ec0c17640afaa15fa50ecb2652e6b2d0f647f1" gracePeriod=30 Oct 30 22:48:26.258596 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.258562 5139 generic.go:356] "Generic (PLEG): container finished" podID="5fa931b0-386c-44c1-91bd-b6ec8940ad26" containerID="cb322c90ff860d63f7058f9635ec0c17640afaa15fa50ecb2652e6b2d0f647f1" exitCode=0 Oct 30 22:48:26.258776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.258647 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" event={"ID":"5fa931b0-386c-44c1-91bd-b6ec8940ad26","Type":"ContainerDied","Data":"cb322c90ff860d63f7058f9635ec0c17640afaa15fa50ecb2652e6b2d0f647f1"} Oct 30 22:48:26.294635 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.294586 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:48:26.407989 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.407963 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa931b0-386c-44c1-91bd-b6ec8940ad26-kube-api-access-shfhz\" (UniqueName: \"kubernetes.io/projected/5fa931b0-386c-44c1-91bd-b6ec8940ad26-kube-api-access-shfhz\") pod \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " Oct 30 22:48:26.408146 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.408008 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls\") pod \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " Oct 30 22:48:26.408229 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.408209 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa931b0-386c-44c1-91bd-b6ec8940ad26-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa931b0-386c-44c1-91bd-b6ec8940ad26-openshift-service-ca\") pod \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " Oct 30 22:48:26.408299 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.408282 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-metrics-certs\") pod \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\" (UID: \"5fa931b0-386c-44c1-91bd-b6ec8940ad26\") " Oct 30 22:48:26.408623 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.408591 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa931b0-386c-44c1-91bd-b6ec8940ad26-openshift-service-ca" pod "5fa931b0-386c-44c1-91bd-b6ec8940ad26" (UID: "5fa931b0-386c-44c1-91bd-b6ec8940ad26"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:26.408825 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.408801 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fa931b0-386c-44c1-91bd-b6ec8940ad26-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:26.411538 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.411486 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa931b0-386c-44c1-91bd-b6ec8940ad26-kube-api-access-shfhz" pod "5fa931b0-386c-44c1-91bd-b6ec8940ad26" (UID: "5fa931b0-386c-44c1-91bd-b6ec8940ad26"). InnerVolumeSpecName "kube-api-access-shfhz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:26.411538 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.411486 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-metrics-certs" pod "5fa931b0-386c-44c1-91bd-b6ec8940ad26" (UID: "5fa931b0-386c-44c1-91bd-b6ec8940ad26"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:26.412537 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.412512 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls" pod "5fa931b0-386c-44c1-91bd-b6ec8940ad26" (UID: "5fa931b0-386c-44c1-91bd-b6ec8940ad26"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:26.510357 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.510318 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:26.510576 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.510374 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shfhz\" (UniqueName: \"kubernetes.io/projected/5fa931b0-386c-44c1-91bd-b6ec8940ad26-kube-api-access-shfhz\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:26.510576 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:26.510389 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fa931b0-386c-44c1-91bd-b6ec8940ad26-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:27.266703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:27.266660 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" event={"ID":"5fa931b0-386c-44c1-91bd-b6ec8940ad26","Type":"ContainerDied","Data":"318319e8844030621f1928305ae13b30818edd868cd72dd581007befa8192d9b"} Oct 30 22:48:27.266703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:27.266706 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv" Oct 30 22:48:27.267231 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:27.266741 5139 scope.go:122] "RemoveContainer" containerID="cb322c90ff860d63f7058f9635ec0c17640afaa15fa50ecb2652e6b2d0f647f1" Oct 30 22:48:27.424979 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:27.424944 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv"] Oct 30 22:48:27.471238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:27.471206 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-8c852b-595f879d58-5gwxv"] Oct 30 22:48:27.546052 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:27.545952 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa931b0-386c-44c1-91bd-b6ec8940ad26" path="/var/lib/kubelet/pods/5fa931b0-386c-44c1-91bd-b6ec8940ad26/volumes" Oct 30 22:48:30.208736 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.208674 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" containerID="cri-o://1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f" gracePeriod=45 Oct 30 22:48:30.476293 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.476233 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 22:48:30.560134 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560099 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-tmp-dir\") pod \"41b1cccb-9e1f-4cae-8142-3072eed003ea\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " Oct 30 22:48:30.560308 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560172 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-home\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-home\") pod \"41b1cccb-9e1f-4cae-8142-3072eed003ea\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " Oct 30 22:48:30.560308 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560202 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-dshm\") pod \"41b1cccb-9e1f-4cae-8142-3072eed003ea\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " Oct 30 22:48:30.560308 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560229 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b1cccb-9e1f-4cae-8142-3072eed003ea-kube-api-access-js8dt\" (UniqueName: \"kubernetes.io/projected/41b1cccb-9e1f-4cae-8142-3072eed003ea-kube-api-access-js8dt\") pod \"41b1cccb-9e1f-4cae-8142-3072eed003ea\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " Oct 30 22:48:30.560541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560312 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-model-cache\") pod \"41b1cccb-9e1f-4cae-8142-3072eed003ea\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " Oct 30 22:48:30.560541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560391 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41b1cccb-9e1f-4cae-8142-3072eed003ea-tls-certs\" (UniqueName: \"kubernetes.io/secret/41b1cccb-9e1f-4cae-8142-3072eed003ea-tls-certs\") pod \"41b1cccb-9e1f-4cae-8142-3072eed003ea\" (UID: \"41b1cccb-9e1f-4cae-8142-3072eed003ea\") " Oct 30 22:48:30.560541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560399 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-tmp-dir" pod "41b1cccb-9e1f-4cae-8142-3072eed003ea" (UID: "41b1cccb-9e1f-4cae-8142-3072eed003ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:30.560541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560459 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-home" pod "41b1cccb-9e1f-4cae-8142-3072eed003ea" (UID: "41b1cccb-9e1f-4cae-8142-3072eed003ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:30.560742 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560582 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-model-cache" pod "41b1cccb-9e1f-4cae-8142-3072eed003ea" (UID: "41b1cccb-9e1f-4cae-8142-3072eed003ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:30.561011 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.560984 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:30.561134 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.561016 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:30.561134 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.561031 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:30.562703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.562673 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-dshm" pod "41b1cccb-9e1f-4cae-8142-3072eed003ea" (UID: "41b1cccb-9e1f-4cae-8142-3072eed003ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:30.562896 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.562880 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41b1cccb-9e1f-4cae-8142-3072eed003ea-tls-certs" pod "41b1cccb-9e1f-4cae-8142-3072eed003ea" (UID: "41b1cccb-9e1f-4cae-8142-3072eed003ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:30.562961 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.562917 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b1cccb-9e1f-4cae-8142-3072eed003ea-kube-api-access-js8dt" pod "41b1cccb-9e1f-4cae-8142-3072eed003ea" (UID: "41b1cccb-9e1f-4cae-8142-3072eed003ea"). InnerVolumeSpecName "kube-api-access-js8dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:30.665144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.662629 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41b1cccb-9e1f-4cae-8142-3072eed003ea-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:30.665144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.662663 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8dt\" (UniqueName: \"kubernetes.io/projected/41b1cccb-9e1f-4cae-8142-3072eed003ea-kube-api-access-js8dt\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:30.665144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:30.662678 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41b1cccb-9e1f-4cae-8142-3072eed003ea-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:31.292271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.292239 5139 generic.go:356] "Generic (PLEG): container finished" podID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerID="1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f" exitCode=0 Oct 30 22:48:31.292709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.292286 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" Oct 30 22:48:31.292709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.292286 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" event={"ID":"41b1cccb-9e1f-4cae-8142-3072eed003ea","Type":"ContainerDied","Data":"1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f"} Oct 30 22:48:31.292709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.292399 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4" event={"ID":"41b1cccb-9e1f-4cae-8142-3072eed003ea","Type":"ContainerDied","Data":"4300c92c4f54684cdb15ffad2f06eeeed586e70dfa451afeafba0c00c92df1d1"} Oct 30 22:48:31.292709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.292436 5139 scope.go:122] "RemoveContainer" containerID="1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f" Oct 30 22:48:31.302724 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.302707 5139 scope.go:122] "RemoveContainer" containerID="1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f" Oct 30 22:48:31.303003 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:48:31.302980 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f\": container with ID starting with 1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f not found: ID does not exist" containerID="1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f" Oct 30 22:48:31.303076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.303016 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f"} err="failed to get container status \"1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f\": rpc error: code = NotFound desc = could not find container \"1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f\": container with ID starting with 1be0c86344aa3a21c86649b5b2bce57e03a508aae2b0552c0d60c97b2625455f not found: ID does not exist" Oct 30 22:48:31.446714 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.446682 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4"] Oct 30 22:48:31.491125 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.491097 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-w1-3c4802-kserve-dc9d5b48-nq2k4"] Oct 30 22:48:31.543967 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.543899 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" path="/var/lib/kubelet/pods/41b1cccb-9e1f-4cae-8142-3072eed003ea/volumes" Oct 30 22:48:31.932719 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.932643 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47"] Oct 30 22:48:31.932881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:31.932860 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" podUID="d291f9ba-26e8-45a5-86c1-91d9807c9e43" containerName="maas-api" containerID="cri-o://6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef" gracePeriod=30 Oct 30 22:48:32.198225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.198169 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:48:32.280077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.280043 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d291f9ba-26e8-45a5-86c1-91d9807c9e43-kube-api-access-x6xfg\" (UniqueName: \"kubernetes.io/projected/d291f9ba-26e8-45a5-86c1-91d9807c9e43-kube-api-access-x6xfg\") pod \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " Oct 30 22:48:32.280253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.280109 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls\") pod \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " Oct 30 22:48:32.280253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.280141 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-metrics-certs\") pod \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " Oct 30 22:48:32.280253 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.280212 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d291f9ba-26e8-45a5-86c1-91d9807c9e43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d291f9ba-26e8-45a5-86c1-91d9807c9e43-openshift-service-ca\") pod \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\" (UID: \"d291f9ba-26e8-45a5-86c1-91d9807c9e43\") " Oct 30 22:48:32.280669 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.280640 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d291f9ba-26e8-45a5-86c1-91d9807c9e43-openshift-service-ca" pod "d291f9ba-26e8-45a5-86c1-91d9807c9e43" (UID: "d291f9ba-26e8-45a5-86c1-91d9807c9e43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:32.280927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.280907 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d291f9ba-26e8-45a5-86c1-91d9807c9e43-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:32.282240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.282217 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls" pod "d291f9ba-26e8-45a5-86c1-91d9807c9e43" (UID: "d291f9ba-26e8-45a5-86c1-91d9807c9e43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:32.282363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.282346 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d291f9ba-26e8-45a5-86c1-91d9807c9e43-kube-api-access-x6xfg" pod "d291f9ba-26e8-45a5-86c1-91d9807c9e43" (UID: "d291f9ba-26e8-45a5-86c1-91d9807c9e43"). InnerVolumeSpecName "kube-api-access-x6xfg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:32.282701 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.282676 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-metrics-certs" pod "d291f9ba-26e8-45a5-86c1-91d9807c9e43" (UID: "d291f9ba-26e8-45a5-86c1-91d9807c9e43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:48:32.301246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.301216 5139 generic.go:356] "Generic (PLEG): container finished" podID="d291f9ba-26e8-45a5-86c1-91d9807c9e43" containerID="6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef" exitCode=0 Oct 30 22:48:32.301654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.301271 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" Oct 30 22:48:32.301654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.301287 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" event={"ID":"d291f9ba-26e8-45a5-86c1-91d9807c9e43","Type":"ContainerDied","Data":"6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef"} Oct 30 22:48:32.301654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.301385 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47" event={"ID":"d291f9ba-26e8-45a5-86c1-91d9807c9e43","Type":"ContainerDied","Data":"d46aabc54ff64c9c1ee871623667573925ee8de984385f94262f6bc5fb8ca395"} Oct 30 22:48:32.301654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.301428 5139 scope.go:122] "RemoveContainer" containerID="6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef" Oct 30 22:48:32.324817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.322011 5139 scope.go:122] "RemoveContainer" containerID="6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef" Oct 30 22:48:32.324817 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:48:32.322469 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef\": container with ID starting with 6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef not found: ID does not exist" containerID="6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef" Oct 30 22:48:32.324817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.322509 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef"} err="failed to get container status \"6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef\": rpc error: code = NotFound desc = could not find container \"6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef\": container with ID starting with 6cf9b32df2b0422673eef3dea677f42efec5a0b96958ee080b2b0019bb65a1ef not found: ID does not exist" Oct 30 22:48:32.382502 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.382469 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6xfg\" (UniqueName: \"kubernetes.io/projected/d291f9ba-26e8-45a5-86c1-91d9807c9e43-kube-api-access-x6xfg\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:32.382502 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.382498 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:32.382502 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.382508 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d291f9ba-26e8-45a5-86c1-91d9807c9e43-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:32.477141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.477063 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47"] Oct 30 22:48:32.481445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:32.481407 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-03e614-5bc5cd7d75-dfz47"] Oct 30 22:48:33.543776 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:33.543741 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d291f9ba-26e8-45a5-86c1-91d9807c9e43" path="/var/lib/kubelet/pods/d291f9ba-26e8-45a5-86c1-91d9807c9e43/volumes" Oct 30 22:48:35.315447 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:35.315385 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk"] Oct 30 22:48:35.316026 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:35.315990 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" containerID="cri-o://faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60" gracePeriod=30 Oct 30 22:48:39.429122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:39.429088 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:48:39.429639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:39.429146 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:48:40.575141 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.575118 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:48:40.673581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673497 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-data\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.673581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673572 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-credential-socket\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.673796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673627 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-envoy\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.673796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673655 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-socket\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.673796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673713 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-podinfo\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.673796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673744 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-certs\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.673796 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673751 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-data" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:40.674076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673846 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istiod-ca-cert\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.674076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673864 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-credential-socket" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:40.674076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673881 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-token\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-token\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.674076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673918 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-kube-api-access-8gfnl\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-kube-api-access-8gfnl\") pod \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\" (UID: \"a92b6c51-1011-4fab-a6d3-929b0f849fc1\") " Oct 30 22:48:40.674076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.673930 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-socket" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:40.674076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.674036 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-certs" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:40.674358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.674168 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istiod-ca-cert" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:40.674448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.674375 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.674448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.674393 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.674448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.674407 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.674448 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.674436 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.674627 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.674449 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.676140 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.676109 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-token" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:40.676267 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.676139 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-kube-api-access-8gfnl" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "kube-api-access-8gfnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:40.676267 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.676258 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-podinfo" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:48:40.676350 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.676280 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-envoy" pod "a92b6c51-1011-4fab-a6d3-929b0f849fc1" (UID: "a92b6c51-1011-4fab-a6d3-929b0f849fc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:40.775628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.775595 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.775628 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.775632 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.775833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.775648 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gfnl\" (UniqueName: \"kubernetes.io/projected/a92b6c51-1011-4fab-a6d3-929b0f849fc1-kube-api-access-8gfnl\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:40.775833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:40.775664 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92b6c51-1011-4fab-a6d3-929b0f849fc1-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:41.187639 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.187603 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc"] Oct 30 22:48:41.188139 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.188107 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" containerID="cri-o://4585d23e0ea020e1b088397d97014911c9c3043ce0119ea76befbce3c6b6ca7c" gracePeriod=30 Oct 30 22:48:41.357505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.357467 5139 generic.go:356] "Generic (PLEG): container finished" podID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerID="faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60" exitCode=0 Oct 30 22:48:41.357676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.357517 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" event={"ID":"a92b6c51-1011-4fab-a6d3-929b0f849fc1","Type":"ContainerDied","Data":"faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60"} Oct 30 22:48:41.357676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.357536 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" Oct 30 22:48:41.357676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.357564 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk" event={"ID":"a92b6c51-1011-4fab-a6d3-929b0f849fc1","Type":"ContainerDied","Data":"9b0c5872a16ce8a4f03cb21779a1c316c029370329469d25f524f125085684b3"} Oct 30 22:48:41.357676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.357590 5139 scope.go:122] "RemoveContainer" containerID="faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60" Oct 30 22:48:41.372823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.372802 5139 scope.go:122] "RemoveContainer" containerID="0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658" Oct 30 22:48:41.381304 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.381289 5139 scope.go:122] "RemoveContainer" containerID="faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60" Oct 30 22:48:41.381581 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:48:41.381559 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60\": container with ID starting with faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60 not found: ID does not exist" containerID="faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60" Oct 30 22:48:41.381620 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.381590 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60"} err="failed to get container status \"faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60\": rpc error: code = NotFound desc = could not find container \"faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60\": container with ID starting with faa0a210a08443e80f22efd44299af603ee06a33bd766d57ae6c1403afd12a60 not found: ID does not exist" Oct 30 22:48:41.381620 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.381608 5139 scope.go:122] "RemoveContainer" containerID="0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658" Oct 30 22:48:41.381817 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:48:41.381797 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658\": container with ID starting with 0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658 not found: ID does not exist" containerID="0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658" Oct 30 22:48:41.381864 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.381822 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658"} err="failed to get container status \"0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658\": rpc error: code = NotFound desc = could not find container \"0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658\": container with ID starting with 0a8955d8bba823555f65714dad53a5c178e532e0903f12666aaf9eac34c29658 not found: ID does not exist" Oct 30 22:48:41.513679 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.513647 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk"] Oct 30 22:48:41.518686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.518661 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c852b-openshift-default-8fbd4dc57-sclzk"] Oct 30 22:48:41.543638 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:41.543612 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" path="/var/lib/kubelet/pods/a92b6c51-1011-4fab-a6d3-929b0f849fc1/volumes" Oct 30 22:48:46.388549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.388518 5139 generic.go:356] "Generic (PLEG): container finished" podID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerID="4585d23e0ea020e1b088397d97014911c9c3043ce0119ea76befbce3c6b6ca7c" exitCode=0 Oct 30 22:48:46.388996 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.388610 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" event={"ID":"ecc489b5-736b-4a87-88fd-e74b5fe32eb2","Type":"ContainerDied","Data":"4585d23e0ea020e1b088397d97014911c9c3043ce0119ea76befbce3c6b6ca7c"} Oct 30 22:48:46.388996 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.388665 5139 scope.go:122] "RemoveContainer" containerID="5bd5e758acceea7eec90439315c7dff118f3c7575331456e4d6de1fb85f0a34a" Oct 30 22:48:46.455278 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.455214 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:48:46.540216 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540186 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-kube-api-access-5s549\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-kube-api-access-5s549\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540225 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-podinfo\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540271 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istiod-ca-cert\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540295 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-socket\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540337 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-envoy\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540376 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-certs\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540483 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-credential-socket\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540511 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-token\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-token\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540601 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-data\") pod \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\" (UID: \"ecc489b5-736b-4a87-88fd-e74b5fe32eb2\") " Oct 30 22:48:46.540753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540654 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-socket" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:46.540981 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540774 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istiod-ca-cert" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:48:46.540981 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.540802 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-credential-socket" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:46.541092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.541003 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-data" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:46.541092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.541037 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-certs" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:46.541268 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.541248 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:46.541358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.541271 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:46.541358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.541287 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:46.541358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.541300 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:46.541358 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.541314 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:46.542786 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.542762 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-kube-api-access-5s549" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "kube-api-access-5s549". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:46.542919 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.542892 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-podinfo" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:48:46.543207 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.543183 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-envoy" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:48:46.543313 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.543204 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-token" pod "ecc489b5-736b-4a87-88fd-e74b5fe32eb2" (UID: "ecc489b5-736b-4a87-88fd-e74b5fe32eb2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:48:46.642778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.642742 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:46.642778 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.642774 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s549\" (UniqueName: \"kubernetes.io/projected/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-kube-api-access-5s549\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:46.642969 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.642788 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:46.642969 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:46.642800 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecc489b5-736b-4a87-88fd-e74b5fe32eb2-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:48:47.395577 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:47.395548 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" Oct 30 22:48:47.395983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:47.395545 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc" event={"ID":"ecc489b5-736b-4a87-88fd-e74b5fe32eb2","Type":"ContainerDied","Data":"964709dc0fc07e8c5ff468f50df9a057d83d200d9593345b021230d96cc94bd1"} Oct 30 22:48:47.395983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:47.395681 5139 scope.go:122] "RemoveContainer" containerID="4585d23e0ea020e1b088397d97014911c9c3043ce0119ea76befbce3c6b6ca7c" Oct 30 22:48:47.599788 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:47.599754 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc"] Oct 30 22:48:47.644880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:47.644853 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-03e614-openshift-default-5c57894957-pd2pc"] Oct 30 22:48:49.542672 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:48:49.542637 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" path="/var/lib/kubelet/pods/ecc489b5-736b-4a87-88fd-e74b5fe32eb2/volumes" Oct 30 22:48:49.804685 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:48:49.804603 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:49:07.588031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:07.588001 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f"] Oct 30 22:49:07.588538 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:07.588211 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" containerID="cri-o://e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387" gracePeriod=30 Oct 30 22:49:12.711357 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.711330 5139 patch_prober.go:29] interesting pod/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" start-of-body= Oct 30 22:49:12.711753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.711380 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.134:15021/healthz/ready\": dial tcp 10.134.0.134:15021: connect: connection refused" Oct 30 22:49:12.849881 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.849852 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:49:12.931337 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931249 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-istio-token\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-istio-token\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.931337 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931301 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-certs\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.931337 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931323 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-data\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.931646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931353 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-credential-socket\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.931646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931382 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-kube-api-access-w94h2\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-kube-api-access-w94h2\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.931646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931447 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-socket\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.931646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931493 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20effb48-4149-4cca-893f-72b67c9cc66a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20effb48-4149-4cca-893f-72b67c9cc66a-istio-podinfo\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.931858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931654 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-envoy\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.931858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931687 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-data" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:12.931858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931706 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-certs" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:12.931858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.931716 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20effb48-4149-4cca-893f-72b67c9cc66a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20effb48-4149-4cca-893f-72b67c9cc66a-istiod-ca-cert\") pod \"20effb48-4149-4cca-893f-72b67c9cc66a\" (UID: \"20effb48-4149-4cca-893f-72b67c9cc66a\") " Oct 30 22:49:12.932245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.932076 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-socket" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:12.932245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.932135 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20effb48-4149-4cca-893f-72b67c9cc66a-istiod-ca-cert" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:12.932245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.932200 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-credential-socket" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:12.932245 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.932235 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:12.932499 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.932254 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:12.932499 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.932267 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:12.933724 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.933698 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-istio-token" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:12.933852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.933760 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-kube-api-access-w94h2" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "kube-api-access-w94h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:12.934047 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.934031 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-envoy" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:49:12.934131 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:12.934117 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20effb48-4149-4cca-893f-72b67c9cc66a-istio-podinfo" pod "20effb48-4149-4cca-893f-72b67c9cc66a" (UID: "20effb48-4149-4cca-893f-72b67c9cc66a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:49:13.033547 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.033514 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.033547 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.033542 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20effb48-4149-4cca-893f-72b67c9cc66a-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.033547 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.033551 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.033800 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.033560 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20effb48-4149-4cca-893f-72b67c9cc66a-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.033800 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.033570 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w94h2\" (UniqueName: \"kubernetes.io/projected/20effb48-4149-4cca-893f-72b67c9cc66a-kube-api-access-w94h2\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.033800 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.033578 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20effb48-4149-4cca-893f-72b67c9cc66a-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:13.549055 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.549017 5139 generic.go:356] "Generic (PLEG): container finished" podID="20effb48-4149-4cca-893f-72b67c9cc66a" containerID="e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387" exitCode=0 Oct 30 22:49:13.549258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.549082 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" Oct 30 22:49:13.549258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.549099 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" event={"ID":"20effb48-4149-4cca-893f-72b67c9cc66a","Type":"ContainerDied","Data":"e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387"} Oct 30 22:49:13.549258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.549145 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f" event={"ID":"20effb48-4149-4cca-893f-72b67c9cc66a","Type":"ContainerDied","Data":"5cacba1f9753ee9522396a19d9d30c8899092b5efa59de8f0c45a652b949d3f5"} Oct 30 22:49:13.549258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.549170 5139 scope.go:122] "RemoveContainer" containerID="e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387" Oct 30 22:49:13.559964 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.559948 5139 scope.go:122] "RemoveContainer" containerID="d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a" Oct 30 22:49:13.568813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.568792 5139 scope.go:122] "RemoveContainer" containerID="e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387" Oct 30 22:49:13.569096 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:49:13.569074 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387\": container with ID starting with e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387 not found: ID does not exist" containerID="e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387" Oct 30 22:49:13.569158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.569104 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387"} err="failed to get container status \"e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387\": rpc error: code = NotFound desc = could not find container \"e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387\": container with ID starting with e69cae43c9f196522353d06dfdf1e44a8752a14c0590f0e6ea6226ca610da387 not found: ID does not exist" Oct 30 22:49:13.569158 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.569122 5139 scope.go:122] "RemoveContainer" containerID="d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a" Oct 30 22:49:13.569372 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:49:13.569356 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a\": container with ID starting with d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a not found: ID does not exist" containerID="d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a" Oct 30 22:49:13.569408 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.569379 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a"} err="failed to get container status \"d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a\": rpc error: code = NotFound desc = could not find container \"d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a\": container with ID starting with d8a4a9863a77227db0cded5fcbd2fff947a383a2de14a30ac4aaad164beb149a not found: ID does not exist" Oct 30 22:49:13.771201 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.771163 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f"] Oct 30 22:49:13.817544 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:13.817465 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-1a5f4a-openshift-default-f4ffc7f85-mgb5f"] Oct 30 22:49:14.477276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477244 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt"] Oct 30 22:49:14.477733 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477718 5139 cpu_manager.go:414] "containerMap: removing container" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" Oct 30 22:49:14.477784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477735 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" Oct 30 22:49:14.477784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477747 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" Oct 30 22:49:14.477784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477752 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" Oct 30 22:49:14.477784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477759 5139 cpu_manager.go:414] "containerMap: removing container" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" Oct 30 22:49:14.477784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477765 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" Oct 30 22:49:14.477784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477771 5139 cpu_manager.go:414] "containerMap: removing container" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" Oct 30 22:49:14.477784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477776 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" Oct 30 22:49:14.477784 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477784 5139 cpu_manager.go:414] "containerMap: removing container" podUID="653cda71-29c3-4cd6-b563-71df8adc1c64" containerName="maas-api" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477789 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653cda71-29c3-4cd6-b563-71df8adc1c64" containerName="maas-api" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477798 5139 cpu_manager.go:414] "containerMap: removing container" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477802 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477812 5139 cpu_manager.go:414] "containerMap: removing container" podUID="d291f9ba-26e8-45a5-86c1-91d9807c9e43" containerName="maas-api" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477816 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d291f9ba-26e8-45a5-86c1-91d9807c9e43" containerName="maas-api" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477824 5139 cpu_manager.go:414] "containerMap: removing container" podUID="0828b8c5-62d8-48ac-b9ac-06dd7373aeb8" containerName="payload-processing" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477828 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0828b8c5-62d8-48ac-b9ac-06dd7373aeb8" containerName="payload-processing" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477841 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c5b51f90-7eec-43b1-af98-3a40d6e66104" containerName="payload-pre-processing" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477845 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5b51f90-7eec-43b1-af98-3a40d6e66104" containerName="payload-pre-processing" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477859 5139 cpu_manager.go:414] "containerMap: removing container" podUID="5fa931b0-386c-44c1-91bd-b6ec8940ad26" containerName="maas-api" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477863 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa931b0-386c-44c1-91bd-b6ec8940ad26" containerName="maas-api" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477871 5139 cpu_manager.go:414] "containerMap: removing container" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477875 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477881 5139 cpu_manager.go:414] "containerMap: removing container" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477886 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477899 5139 cpu_manager.go:414] "containerMap: removing container" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477903 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477973 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="0828b8c5-62d8-48ac-b9ac-06dd7373aeb8" containerName="payload-processing" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477982 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5b51f90-7eec-43b1-af98-3a40d6e66104" containerName="payload-pre-processing" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477989 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa931b0-386c-44c1-91bd-b6ec8940ad26" containerName="maas-api" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.477995 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478001 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e8093e-00ea-4e68-8923-d3556d1ae1e3" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478007 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478012 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478016 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478023 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b1cccb-9e1f-4cae-8142-3072eed003ea" containerName="main" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478029 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478034 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478039 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" Oct 30 22:49:14.478031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478044 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" Oct 30 22:49:14.478927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478049 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="653cda71-29c3-4cd6-b563-71df8adc1c64" containerName="maas-api" Oct 30 22:49:14.478927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478054 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" containerName="istio-proxy" Oct 30 22:49:14.478927 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.478061 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="d291f9ba-26e8-45a5-86c1-91d9807c9e43" containerName="maas-api" Oct 30 22:49:14.482786 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.482766 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.533995 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.533969 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-0c4786-serving-cert\"" Oct 30 22:49:14.550403 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.550371 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63a8686-d9b5-40a7-8872-4ca63ddd1005-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.550565 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.550535 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.550618 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.550563 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p26dz\" (UniqueName: \"kubernetes.io/projected/d63a8686-d9b5-40a7-8872-4ca63ddd1005-kube-api-access-p26dz\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.550691 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.550667 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-metrics-certs\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.588957 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.588932 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt"] Oct 30 22:49:14.651966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.651936 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.652144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.651970 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p26dz\" (UniqueName: \"kubernetes.io/projected/d63a8686-d9b5-40a7-8872-4ca63ddd1005-kube-api-access-p26dz\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.652144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.652015 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-metrics-certs\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.652144 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:49:14.652079 5139 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-serving-cert: secret "maas-api-e2e-extmod-a-0c4786-serving-cert" not found Oct 30 22:49:14.652144 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.652115 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63a8686-d9b5-40a7-8872-4ca63ddd1005-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.652384 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:49:14.652158 5139 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls podName:d63a8686-d9b5-40a7-8872-4ca63ddd1005 nodeName:}" failed. No retries permitted until 2026-10-30 22:49:15.152138353 +0000 UTC m=+4263.743423846 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls") pod "maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" (UID: "d63a8686-d9b5-40a7-8872-4ca63ddd1005") : secret "maas-api-e2e-extmod-a-0c4786-serving-cert" not found Oct 30 22:49:14.652837 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.652817 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63a8686-d9b5-40a7-8872-4ca63ddd1005-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.654350 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.654331 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-metrics-certs\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:14.711889 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:14.711859 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p26dz\" (UniqueName: \"kubernetes.io/projected/d63a8686-d9b5-40a7-8872-4ca63ddd1005-kube-api-access-p26dz\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:15.158782 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:15.158741 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:15.161277 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:15.161253 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls\") pod \"maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:15.396637 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:15.396598 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:15.543520 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:15.543489 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20effb48-4149-4cca-893f-72b67c9cc66a" path="/var/lib/kubelet/pods/20effb48-4149-4cca-893f-72b67c9cc66a/volumes" Oct 30 22:49:15.775787 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:15.775755 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt"] Oct 30 22:49:15.780777 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:49:15.780740 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63a8686_d9b5_40a7_8872_4ca63ddd1005.slice/crio-8d36d79fa068c86997a6a4525375ce4c1d0805e876ddc71382546370a5d4a758 WatchSource:0}: Error finding container 8d36d79fa068c86997a6a4525375ce4c1d0805e876ddc71382546370a5d4a758: Status 404 returned error can't find the container with id 8d36d79fa068c86997a6a4525375ce4c1d0805e876ddc71382546370a5d4a758 Oct 30 22:49:16.567855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:16.567819 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" event={"ID":"d63a8686-d9b5-40a7-8872-4ca63ddd1005","Type":"ContainerStarted","Data":"8d36d79fa068c86997a6a4525375ce4c1d0805e876ddc71382546370a5d4a758"} Oct 30 22:49:17.574285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:17.574244 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" event={"ID":"d63a8686-d9b5-40a7-8872-4ca63ddd1005","Type":"ContainerStarted","Data":"9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace"} Oct 30 22:49:17.574689 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:17.574305 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:17.690908 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:17.690852 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" podStartSLOduration=2.66027475 podStartE2EDuration="3.690838608s" totalImagesPullingTime="1.030563858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:17.68910811 +0000 UTC m=+4266.280393596" watchObservedRunningTime="2026-10-30 22:49:17.690838608 +0000 UTC m=+4266.282124095" Oct 30 22:49:19.806454 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:49:19.806399 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:49:23.585876 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:23.585849 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:49:46.461100 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.461066 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56"] Oct 30 22:49:46.461689 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.461255 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" podUID="09bccc97-05ea-4d94-8887-c0d6f1e3bd19" containerName="maas-api" containerID="cri-o://958dba0cfaaeba443a3420772f1c79fd738e523afd1ec350e84ac0be8705e72b" gracePeriod=30 Oct 30 22:49:46.729839 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.729809 5139 generic.go:356] "Generic (PLEG): container finished" podID="09bccc97-05ea-4d94-8887-c0d6f1e3bd19" containerID="958dba0cfaaeba443a3420772f1c79fd738e523afd1ec350e84ac0be8705e72b" exitCode=0 Oct 30 22:49:46.729990 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.729860 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" event={"ID":"09bccc97-05ea-4d94-8887-c0d6f1e3bd19","Type":"ContainerDied","Data":"958dba0cfaaeba443a3420772f1c79fd738e523afd1ec350e84ac0be8705e72b"} Oct 30 22:49:46.729990 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.729900 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" event={"ID":"09bccc97-05ea-4d94-8887-c0d6f1e3bd19","Type":"ContainerDied","Data":"cc3715b1496c0639363dcf45caf9e2edac5a8021de27b938a1619d143eb58946"} Oct 30 22:49:46.729990 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.729916 5139 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc3715b1496c0639363dcf45caf9e2edac5a8021de27b938a1619d143eb58946" Oct 30 22:49:46.731371 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.731353 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:49:46.814900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.814870 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls\") pod \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " Oct 30 22:49:46.815067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.814935 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-metrics-certs\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-metrics-certs\") pod \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " Oct 30 22:49:46.815067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.814994 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-kube-api-access-gn98j\" (UniqueName: \"kubernetes.io/projected/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-kube-api-access-gn98j\") pod \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " Oct 30 22:49:46.815067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.815023 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-openshift-service-ca\") pod \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\" (UID: \"09bccc97-05ea-4d94-8887-c0d6f1e3bd19\") " Oct 30 22:49:46.815495 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.815464 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-openshift-service-ca" pod "09bccc97-05ea-4d94-8887-c0d6f1e3bd19" (UID: "09bccc97-05ea-4d94-8887-c0d6f1e3bd19"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:46.817025 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.816998 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls" pod "09bccc97-05ea-4d94-8887-c0d6f1e3bd19" (UID: "09bccc97-05ea-4d94-8887-c0d6f1e3bd19"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:46.817120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.817047 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-metrics-certs" pod "09bccc97-05ea-4d94-8887-c0d6f1e3bd19" (UID: "09bccc97-05ea-4d94-8887-c0d6f1e3bd19"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:46.817120 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.817066 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-kube-api-access-gn98j" pod "09bccc97-05ea-4d94-8887-c0d6f1e3bd19" (UID: "09bccc97-05ea-4d94-8887-c0d6f1e3bd19"). InnerVolumeSpecName "kube-api-access-gn98j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:46.916024 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.915991 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:46.916024 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.916018 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:46.916024 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.916028 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn98j\" (UniqueName: \"kubernetes.io/projected/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-kube-api-access-gn98j\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:46.916263 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:46.916038 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/09bccc97-05ea-4d94-8887-c0d6f1e3bd19-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:47.563555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.563524 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2"] Oct 30 22:49:47.563904 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.563752 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" podUID="baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" containerName="maas-api" containerID="cri-o://cb9ba216219088bee6b569fa31c871bcdfb500fce4e7f67a48ab8eb791e9f0ce" gracePeriod=30 Oct 30 22:49:47.738008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.737729 5139 generic.go:356] "Generic (PLEG): container finished" podID="baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" containerID="cb9ba216219088bee6b569fa31c871bcdfb500fce4e7f67a48ab8eb791e9f0ce" exitCode=0 Oct 30 22:49:47.738008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.737813 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" event={"ID":"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4","Type":"ContainerDied","Data":"cb9ba216219088bee6b569fa31c871bcdfb500fce4e7f67a48ab8eb791e9f0ce"} Oct 30 22:49:47.738008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.737971 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56" Oct 30 22:49:47.825609 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.825582 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 22:49:47.892745 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.892714 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56"] Oct 30 22:49:47.898911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.898887 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-7a134d-5fc5f4c5c9-5fk56"] Oct 30 22:49:47.926438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.926393 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-maas-api-tls\") pod \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " Oct 30 22:49:47.926580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.926449 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-openshift-service-ca\") pod \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " Oct 30 22:49:47.926580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.926531 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-kube-api-access-sxpgl\" (UniqueName: \"kubernetes.io/projected/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-kube-api-access-sxpgl\") pod \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " Oct 30 22:49:47.926663 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.926578 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-metrics-certs\") pod \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\" (UID: \"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4\") " Oct 30 22:49:47.926858 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.926826 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-openshift-service-ca" pod "baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" (UID: "baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:47.926973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.926959 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:47.928646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.928623 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-kube-api-access-sxpgl" pod "baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" (UID: "baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4"). InnerVolumeSpecName "kube-api-access-sxpgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:47.928646 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.928640 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-metrics-certs" pod "baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" (UID: "baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:47.928793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:47.928694 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-maas-api-tls" pod "baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" (UID: "baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:48.028012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.027973 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxpgl\" (UniqueName: \"kubernetes.io/projected/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-kube-api-access-sxpgl\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:48.028012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.028005 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:48.028012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.028016 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:48.685194 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.685147 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd"] Oct 30 22:49:48.685707 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.685428 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" containerID="cri-o://7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff" gracePeriod=30 Oct 30 22:49:48.747906 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.747870 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" Oct 30 22:49:48.747906 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.747894 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2" event={"ID":"baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4","Type":"ContainerDied","Data":"65bf8fb40d61c96719ca89716fa3d53ecefd49463c854a50f8ebe9fb04de6c80"} Oct 30 22:49:48.748181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.747949 5139 scope.go:122] "RemoveContainer" containerID="cb9ba216219088bee6b569fa31c871bcdfb500fce4e7f67a48ab8eb791e9f0ce" Oct 30 22:49:48.905195 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.905162 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2"] Oct 30 22:49:48.915955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.915912 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-3c4802-697c94f6bd-dq8d2"] Oct 30 22:49:48.955293 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:48.955229 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:49:49.039297 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.039264 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-metrics-certs\") pod \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " Oct 30 22:49:49.039496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.039375 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/772c18d4-c0b2-4aa7-952d-25e11e4a945f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772c18d4-c0b2-4aa7-952d-25e11e4a945f-openshift-service-ca\") pod \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " Oct 30 22:49:49.039496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.039409 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls\") pod \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " Oct 30 22:49:49.039496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.039471 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/772c18d4-c0b2-4aa7-952d-25e11e4a945f-kube-api-access-vpfd6\" (UniqueName: \"kubernetes.io/projected/772c18d4-c0b2-4aa7-952d-25e11e4a945f-kube-api-access-vpfd6\") pod \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\" (UID: \"772c18d4-c0b2-4aa7-952d-25e11e4a945f\") " Oct 30 22:49:49.039731 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.039707 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/772c18d4-c0b2-4aa7-952d-25e11e4a945f-openshift-service-ca" pod "772c18d4-c0b2-4aa7-952d-25e11e4a945f" (UID: "772c18d4-c0b2-4aa7-952d-25e11e4a945f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:49:49.039862 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.039843 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/772c18d4-c0b2-4aa7-952d-25e11e4a945f-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:49.041554 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.041525 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/772c18d4-c0b2-4aa7-952d-25e11e4a945f-kube-api-access-vpfd6" pod "772c18d4-c0b2-4aa7-952d-25e11e4a945f" (UID: "772c18d4-c0b2-4aa7-952d-25e11e4a945f"). InnerVolumeSpecName "kube-api-access-vpfd6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:49:49.041554 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.041543 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls" pod "772c18d4-c0b2-4aa7-952d-25e11e4a945f" (UID: "772c18d4-c0b2-4aa7-952d-25e11e4a945f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:49.041706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.041558 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-metrics-certs" pod "772c18d4-c0b2-4aa7-952d-25e11e4a945f" (UID: "772c18d4-c0b2-4aa7-952d-25e11e4a945f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:49:49.141094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.141059 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:49.141094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.141094 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpfd6\" (UniqueName: \"kubernetes.io/projected/772c18d4-c0b2-4aa7-952d-25e11e4a945f-kube-api-access-vpfd6\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:49.141300 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.141107 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772c18d4-c0b2-4aa7-952d-25e11e4a945f-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:49:49.542819 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.542785 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bccc97-05ea-4d94-8887-c0d6f1e3bd19" path="/var/lib/kubelet/pods/09bccc97-05ea-4d94-8887-c0d6f1e3bd19/volumes" Oct 30 22:49:49.543247 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.543234 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" path="/var/lib/kubelet/pods/baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4/volumes" Oct 30 22:49:49.755549 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.755515 5139 generic.go:356] "Generic (PLEG): container finished" podID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerID="7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff" exitCode=0 Oct 30 22:49:49.755960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.755580 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" Oct 30 22:49:49.755960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.755594 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" event={"ID":"772c18d4-c0b2-4aa7-952d-25e11e4a945f","Type":"ContainerDied","Data":"7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff"} Oct 30 22:49:49.755960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.755650 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" event={"ID":"772c18d4-c0b2-4aa7-952d-25e11e4a945f","Type":"ContainerDied","Data":"c15075c23754fc0fe921e5abd9e9087cae85c979146e0060f6cba545b770149d"} Oct 30 22:49:49.755960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.755675 5139 scope.go:122] "RemoveContainer" containerID="7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff" Oct 30 22:49:49.767350 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.767321 5139 scope.go:122] "RemoveContainer" containerID="3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96" Oct 30 22:49:49.775994 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.775973 5139 scope.go:122] "RemoveContainer" containerID="7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff" Oct 30 22:49:49.776306 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:49:49.776287 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff\": container with ID starting with 7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff not found: ID does not exist" containerID="7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff" Oct 30 22:49:49.776363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.776316 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff"} err="failed to get container status \"7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff\": rpc error: code = NotFound desc = could not find container \"7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff\": container with ID starting with 7e91fc4327189df9a7ad8a4c0c8b0c3915ee9ddaa83a5299132c4bb85e2640ff not found: ID does not exist" Oct 30 22:49:49.776363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.776337 5139 scope.go:122] "RemoveContainer" containerID="3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96" Oct 30 22:49:49.776611 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:49:49.776595 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96\": container with ID starting with 3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96 not found: ID does not exist" containerID="3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96" Oct 30 22:49:49.776669 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.776617 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96"} err="failed to get container status \"3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96\": rpc error: code = NotFound desc = could not find container \"3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96\": container with ID starting with 3a1f7b2c4a8d03b0375e80e0ab3f4b9d0f7ecdfa61a12093bc59a365b25c1e96 not found: ID does not exist" Oct 30 22:49:49.803092 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:49:49.803027 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:49:49.910206 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.910178 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd"] Oct 30 22:49:49.956587 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:49.956558 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd"] Oct 30 22:49:51.545204 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:51.545170 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" path="/var/lib/kubelet/pods/772c18d4-c0b2-4aa7-952d-25e11e4a945f/volumes" Oct 30 22:49:51.892398 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:51.892315 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2cc983-7895d56db7-p7vqd" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.136:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 22:49:59.128750 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:59.128713 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp"] Oct 30 22:49:59.129202 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:49:59.129100 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" containerID="cri-o://fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d" gracePeriod=30 Oct 30 22:50:04.388325 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.388295 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:50:04.512643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.512607 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-certs\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.512643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.512644 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-podinfo\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.512922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.512696 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-token\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-token\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.512922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.512739 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-credential-socket\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.512922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.512836 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-data\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.512922 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.512911 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-certs" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:04.513140 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.512946 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-socket\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.513140 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513006 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-envoy\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.513140 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513031 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-credential-socket" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:04.513140 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513115 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc58be14-16c1-4d36-a08e-9736d0c862dd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc58be14-16c1-4d36-a08e-9736d0c862dd-istiod-ca-cert\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.513347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513156 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-data" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:04.513347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513212 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-socket" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:04.513347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513205 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-kube-api-access-cxg2k\") pod \"cc58be14-16c1-4d36-a08e-9736d0c862dd\" (UID: \"cc58be14-16c1-4d36-a08e-9736d0c862dd\") " Oct 30 22:50:04.513576 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513542 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc58be14-16c1-4d36-a08e-9736d0c862dd-istiod-ca-cert" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:50:04.513793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513722 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.513793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513744 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.513793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513759 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.513793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513772 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc58be14-16c1-4d36-a08e-9736d0c862dd-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.513793 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.513783 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.515291 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.515253 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-envoy" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:04.515566 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.515544 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-podinfo" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:50:04.515566 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.515556 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-kube-api-access-cxg2k" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "kube-api-access-cxg2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:04.515688 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.515669 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-token" pod "cc58be14-16c1-4d36-a08e-9736d0c862dd" (UID: "cc58be14-16c1-4d36-a08e-9736d0c862dd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:04.601275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.601240 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6"] Oct 30 22:50:04.604783 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.603852 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl"] Oct 30 22:50:04.615114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.615080 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.615114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.615108 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-kube-api-access-cxg2k\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.615114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.615122 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.615408 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.615131 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc58be14-16c1-4d36-a08e-9736d0c862dd-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:04.837034 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.836940 5139 generic.go:356] "Generic (PLEG): container finished" podID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerID="fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d" exitCode=0 Oct 30 22:50:04.837034 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.836990 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" Oct 30 22:50:04.837034 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.837009 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" event={"ID":"cc58be14-16c1-4d36-a08e-9736d0c862dd","Type":"ContainerDied","Data":"fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d"} Oct 30 22:50:04.837273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.837053 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp" event={"ID":"cc58be14-16c1-4d36-a08e-9736d0c862dd","Type":"ContainerDied","Data":"ba28203d99468cd037550e17837e83393181835023403509bf9f30bf2962a63d"} Oct 30 22:50:04.837273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.837074 5139 scope.go:122] "RemoveContainer" containerID="fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d" Oct 30 22:50:04.846920 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.846901 5139 scope.go:122] "RemoveContainer" containerID="fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec" Oct 30 22:50:04.855283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.855259 5139 scope.go:122] "RemoveContainer" containerID="fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d" Oct 30 22:50:04.855579 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:50:04.855558 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d\": container with ID starting with fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d not found: ID does not exist" containerID="fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d" Oct 30 22:50:04.855654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.855587 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d"} err="failed to get container status \"fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d\": rpc error: code = NotFound desc = could not find container \"fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d\": container with ID starting with fb4b177f270785f1822780d614b23f400d3e22a89c5e9e6a2c160bcb1e9c848d not found: ID does not exist" Oct 30 22:50:04.855654 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.855607 5139 scope.go:122] "RemoveContainer" containerID="fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec" Oct 30 22:50:04.855871 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:50:04.855852 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec\": container with ID starting with fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec not found: ID does not exist" containerID="fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec" Oct 30 22:50:04.855913 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.855880 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec"} err="failed to get container status \"fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec\": rpc error: code = NotFound desc = could not find container \"fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec\": container with ID starting with fc6e93cb6480b701e023f94c35d2bd66db6ce637d445d53be1bdd10327de1eec not found: ID does not exist" Oct 30 22:50:04.992535 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.992502 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp"] Oct 30 22:50:04.996092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:04.996063 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-2cc983-openshift-default-649c558565-vwjfp"] Oct 30 22:50:05.542958 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:05.542917 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" path="/var/lib/kubelet/pods/cc58be14-16c1-4d36-a08e-9736d0c862dd/volumes" Oct 30 22:50:07.933164 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:07.933129 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng"] Oct 30 22:50:09.647265 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.647224 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25"] Oct 30 22:50:09.647756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.647504 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" podUID="17c8ec1b-452e-49b5-8708-4093b854ef9b" containerName="maas-api" containerID="cri-o://b34c3490a15bf4c90454401574745187b48df949e1807a18d25471a64ccc0567" gracePeriod=30 Oct 30 22:50:09.867911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.867861 5139 generic.go:356] "Generic (PLEG): container finished" podID="17c8ec1b-452e-49b5-8708-4093b854ef9b" containerID="b34c3490a15bf4c90454401574745187b48df949e1807a18d25471a64ccc0567" exitCode=0 Oct 30 22:50:09.868240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.867934 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" event={"ID":"17c8ec1b-452e-49b5-8708-4093b854ef9b","Type":"ContainerDied","Data":"b34c3490a15bf4c90454401574745187b48df949e1807a18d25471a64ccc0567"} Oct 30 22:50:09.908850 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.908795 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:50:09.975400 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.975368 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17c8ec1b-452e-49b5-8708-4093b854ef9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8ec1b-452e-49b5-8708-4093b854ef9b-openshift-service-ca\") pod \"17c8ec1b-452e-49b5-8708-4093b854ef9b\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " Oct 30 22:50:09.975598 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.975405 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls\") pod \"17c8ec1b-452e-49b5-8708-4093b854ef9b\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " Oct 30 22:50:09.975598 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.975482 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17c8ec1b-452e-49b5-8708-4093b854ef9b-kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/17c8ec1b-452e-49b5-8708-4093b854ef9b-kube-api-access-qdfxh\") pod \"17c8ec1b-452e-49b5-8708-4093b854ef9b\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " Oct 30 22:50:09.975598 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.975568 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-metrics-certs\") pod \"17c8ec1b-452e-49b5-8708-4093b854ef9b\" (UID: \"17c8ec1b-452e-49b5-8708-4093b854ef9b\") " Oct 30 22:50:09.975774 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.975749 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17c8ec1b-452e-49b5-8708-4093b854ef9b-openshift-service-ca" pod "17c8ec1b-452e-49b5-8708-4093b854ef9b" (UID: "17c8ec1b-452e-49b5-8708-4093b854ef9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:50:09.975972 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.975950 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17c8ec1b-452e-49b5-8708-4093b854ef9b-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:09.977711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.977684 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-metrics-certs" pod "17c8ec1b-452e-49b5-8708-4093b854ef9b" (UID: "17c8ec1b-452e-49b5-8708-4093b854ef9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:09.977711 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.977709 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls" pod "17c8ec1b-452e-49b5-8708-4093b854ef9b" (UID: "17c8ec1b-452e-49b5-8708-4093b854ef9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:09.977868 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:09.977740 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17c8ec1b-452e-49b5-8708-4093b854ef9b-kube-api-access-qdfxh" pod "17c8ec1b-452e-49b5-8708-4093b854ef9b" (UID: "17c8ec1b-452e-49b5-8708-4093b854ef9b"). InnerVolumeSpecName "kube-api-access-qdfxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:10.077212 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:10.077179 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:10.077212 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:10.077210 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdfxh\" (UniqueName: \"kubernetes.io/projected/17c8ec1b-452e-49b5-8708-4093b854ef9b-kube-api-access-qdfxh\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:10.077212 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:10.077220 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17c8ec1b-452e-49b5-8708-4093b854ef9b-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:10.875285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:10.875258 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" Oct 30 22:50:10.875709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:10.875255 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25" event={"ID":"17c8ec1b-452e-49b5-8708-4093b854ef9b","Type":"ContainerDied","Data":"85d07f7188fc488432a07ddbb140e1ea9c4e5b5f3207fcb1d221e89f83431086"} Oct 30 22:50:10.875709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:10.875389 5139 scope.go:122] "RemoveContainer" containerID="b34c3490a15bf4c90454401574745187b48df949e1807a18d25471a64ccc0567" Oct 30 22:50:11.038109 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:11.038076 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25"] Oct 30 22:50:11.084983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:11.084950 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-b-6b46c3-869f58cc87-txz25"] Oct 30 22:50:11.543492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:11.543452 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17c8ec1b-452e-49b5-8708-4093b854ef9b" path="/var/lib/kubelet/pods/17c8ec1b-452e-49b5-8708-4093b854ef9b/volumes" Oct 30 22:50:12.934617 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:12.934566 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" podUID="4a0e33b4-d4e4-468f-b9b4-c78654a41f13" containerName="payload-processing" containerID="cri-o://6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a" gracePeriod=25 Oct 30 22:50:16.570966 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:16.570928 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" podUID="4a0e33b4-d4e4-468f-b9b4-c78654a41f13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:50:19.625174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.625131 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" containerID="cri-o://dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da" gracePeriod=45 Oct 30 22:50:19.628438 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.628391 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" podUID="1dbb1410-3b3b-4a28-bb15-92efac37aa44" containerName="main" containerID="cri-o://af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5" gracePeriod=45 Oct 30 22:50:19.750763 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.750723 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" podUID="1dbb1410-3b3b-4a28-bb15-92efac37aa44" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": dial tcp 10.134.0.74:8000: connect: connection refused" Oct 30 22:50:19.815088 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:50:19.814973 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:50:19.887740 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.887677 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 22:50:19.914041 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.914018 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:50:19.927871 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.927839 5139 generic.go:356] "Generic (PLEG): container finished" podID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerID="dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da" exitCode=0 Oct 30 22:50:19.928016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.927883 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" Oct 30 22:50:19.928016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.927927 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" event={"ID":"90a132e5-9eef-4f67-a14f-5d74e05f4e75","Type":"ContainerDied","Data":"dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da"} Oct 30 22:50:19.928016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.927967 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" event={"ID":"90a132e5-9eef-4f67-a14f-5d74e05f4e75","Type":"ContainerDied","Data":"b2b57a4d2a6d16eab9f9c54af3873d4a682a21cc4de0fe890afa3dc267ae43bf"} Oct 30 22:50:19.928016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.927994 5139 scope.go:122] "RemoveContainer" containerID="dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da" Oct 30 22:50:19.929367 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.929343 5139 generic.go:356] "Generic (PLEG): container finished" podID="1dbb1410-3b3b-4a28-bb15-92efac37aa44" containerID="af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5" exitCode=0 Oct 30 22:50:19.929492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.929386 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" Oct 30 22:50:19.929492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.929403 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" event={"ID":"1dbb1410-3b3b-4a28-bb15-92efac37aa44","Type":"ContainerDied","Data":"af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5"} Oct 30 22:50:19.929492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.929455 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6" event={"ID":"1dbb1410-3b3b-4a28-bb15-92efac37aa44","Type":"ContainerDied","Data":"8b7a42080a1e33b3b6d63028ddb5cf0832d78871a3aaa0280cd6b404295d9e2e"} Oct 30 22:50:19.936683 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.936663 5139 scope.go:122] "RemoveContainer" containerID="dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da" Oct 30 22:50:19.936956 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:50:19.936932 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da\": container with ID starting with dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da not found: ID does not exist" containerID="dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da" Oct 30 22:50:19.937023 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.936964 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da"} err="failed to get container status \"dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da\": rpc error: code = NotFound desc = could not find container \"dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da\": container with ID starting with dd008c6ddbb704642efcbfeff980974629a27949b22b1ff49abcee48fec925da not found: ID does not exist" Oct 30 22:50:19.937023 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.936981 5139 scope.go:122] "RemoveContainer" containerID="af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5" Oct 30 22:50:19.945388 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.945370 5139 scope.go:122] "RemoveContainer" containerID="af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5" Oct 30 22:50:19.945666 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:50:19.945646 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5\": container with ID starting with af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5 not found: ID does not exist" containerID="af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5" Oct 30 22:50:19.945730 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.945675 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5"} err="failed to get container status \"af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5\": rpc error: code = NotFound desc = could not find container \"af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5\": container with ID starting with af7b79ba2147f6c262bf8562a2d938f8409f78a52c46393c467f7f6835f08dd5 not found: ID does not exist" Oct 30 22:50:19.985016 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.984981 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbb1410-3b3b-4a28-bb15-92efac37aa44-kube-api-access-z2jxl\" (UniqueName: \"kubernetes.io/projected/1dbb1410-3b3b-4a28-bb15-92efac37aa44-kube-api-access-z2jxl\") pod \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " Oct 30 22:50:19.985173 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985045 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tls-certs\" (UniqueName: \"kubernetes.io/secret/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tls-certs\") pod \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " Oct 30 22:50:19.985173 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985082 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tmp-dir\") pod \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " Oct 30 22:50:19.985173 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985128 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90a132e5-9eef-4f67-a14f-5d74e05f4e75-kube-api-access-nt6lb\" (UniqueName: \"kubernetes.io/projected/90a132e5-9eef-4f67-a14f-5d74e05f4e75-kube-api-access-nt6lb\") pod \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " Oct 30 22:50:19.985173 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985160 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-dshm\") pod \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " Oct 30 22:50:19.985387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985180 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tls-certs\" (UniqueName: \"kubernetes.io/secret/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tls-certs\") pod \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " Oct 30 22:50:19.985387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985207 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-home\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-home\") pod \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " Oct 30 22:50:19.985387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985311 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tmp-dir\") pod \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " Oct 30 22:50:19.985387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985340 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-home\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-home\") pod \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " Oct 30 22:50:19.985387 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985374 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-model-cache\") pod \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\" (UID: \"1dbb1410-3b3b-4a28-bb15-92efac37aa44\") " Oct 30 22:50:19.985686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985464 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-dshm\") pod \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " Oct 30 22:50:19.985686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985484 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tmp-dir" pod "1dbb1410-3b3b-4a28-bb15-92efac37aa44" (UID: "1dbb1410-3b3b-4a28-bb15-92efac37aa44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:19.985686 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985502 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-model-cache\") pod \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\" (UID: \"90a132e5-9eef-4f67-a14f-5d74e05f4e75\") " Oct 30 22:50:19.985846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985702 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-home" pod "1dbb1410-3b3b-4a28-bb15-92efac37aa44" (UID: "1dbb1410-3b3b-4a28-bb15-92efac37aa44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:19.985846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985809 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-home" pod "90a132e5-9eef-4f67-a14f-5d74e05f4e75" (UID: "90a132e5-9eef-4f67-a14f-5d74e05f4e75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:19.985946 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.985919 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-model-cache" pod "90a132e5-9eef-4f67-a14f-5d74e05f4e75" (UID: "90a132e5-9eef-4f67-a14f-5d74e05f4e75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:19.986029 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.986006 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-model-cache" pod "1dbb1410-3b3b-4a28-bb15-92efac37aa44" (UID: "1dbb1410-3b3b-4a28-bb15-92efac37aa44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:19.986094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.986035 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:19.986094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.986056 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:19.986094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.986068 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:19.986094 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.986081 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:19.986257 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.986191 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tmp-dir" pod "90a132e5-9eef-4f67-a14f-5d74e05f4e75" (UID: "90a132e5-9eef-4f67-a14f-5d74e05f4e75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:19.988054 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.988028 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90a132e5-9eef-4f67-a14f-5d74e05f4e75-kube-api-access-nt6lb" pod "90a132e5-9eef-4f67-a14f-5d74e05f4e75" (UID: "90a132e5-9eef-4f67-a14f-5d74e05f4e75"). InnerVolumeSpecName "kube-api-access-nt6lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:19.988054 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.988040 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbb1410-3b3b-4a28-bb15-92efac37aa44-kube-api-access-z2jxl" pod "1dbb1410-3b3b-4a28-bb15-92efac37aa44" (UID: "1dbb1410-3b3b-4a28-bb15-92efac37aa44"). InnerVolumeSpecName "kube-api-access-z2jxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:19.988186 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.988071 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-dshm" pod "90a132e5-9eef-4f67-a14f-5d74e05f4e75" (UID: "90a132e5-9eef-4f67-a14f-5d74e05f4e75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:19.988186 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.988074 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tls-certs" pod "90a132e5-9eef-4f67-a14f-5d74e05f4e75" (UID: "90a132e5-9eef-4f67-a14f-5d74e05f4e75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:19.988186 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.988087 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-dshm" pod "1dbb1410-3b3b-4a28-bb15-92efac37aa44" (UID: "1dbb1410-3b3b-4a28-bb15-92efac37aa44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:50:19.988298 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:19.988271 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tls-certs" pod "1dbb1410-3b3b-4a28-bb15-92efac37aa44" (UID: "1dbb1410-3b3b-4a28-bb15-92efac37aa44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:50:20.087636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.087601 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:20.087636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.087637 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt6lb\" (UniqueName: \"kubernetes.io/projected/90a132e5-9eef-4f67-a14f-5d74e05f4e75-kube-api-access-nt6lb\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:20.087846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.087651 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:20.087846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.087665 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1dbb1410-3b3b-4a28-bb15-92efac37aa44-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:20.087846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.087678 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:20.087846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.087693 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1dbb1410-3b3b-4a28-bb15-92efac37aa44-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:20.087846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.087707 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/90a132e5-9eef-4f67-a14f-5d74e05f4e75-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:20.087846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.087721 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2jxl\" (UniqueName: \"kubernetes.io/projected/1dbb1410-3b3b-4a28-bb15-92efac37aa44-kube-api-access-z2jxl\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:20.344062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.344027 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6"] Oct 30 22:50:20.348713 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.348684 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/distinct-2-w4-26743a-kserve-76599f66b5-9kph6"] Oct 30 22:50:20.455870 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.455836 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl"] Oct 30 22:50:20.466442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.461659 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl"] Oct 30 22:50:20.858284 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:20.858241 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/premium-w4-26743a-kserve-7b8c7bb9b7-92vtl" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 22:50:21.543874 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:21.543843 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbb1410-3b3b-4a28-bb15-92efac37aa44" path="/var/lib/kubelet/pods/1dbb1410-3b3b-4a28-bb15-92efac37aa44/volumes" Oct 30 22:50:21.544244 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:21.544231 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" path="/var/lib/kubelet/pods/90a132e5-9eef-4f67-a14f-5d74e05f4e75/volumes" Oct 30 22:50:23.689433 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689396 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl"] Oct 30 22:50:23.689835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689818 5139 cpu_manager.go:414] "containerMap: removing container" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" Oct 30 22:50:23.689880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689835 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4f44d4-a36e-4779-b7ab-908329d4691b" containerName="istio-proxy" Oct 30 22:50:23.689880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689847 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" Oct 30 22:50:23.689880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689852 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc489b5-736b-4a87-88fd-e74b5fe32eb2" containerName="istio-proxy" Oct 30 22:50:23.689880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689858 5139 cpu_manager.go:414] "containerMap: removing container" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" Oct 30 22:50:23.689880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689863 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92b6c51-1011-4fab-a6d3-929b0f849fc1" containerName="istio-proxy" Oct 30 22:50:23.689880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689874 5139 cpu_manager.go:414] "containerMap: removing container" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" Oct 30 22:50:23.689880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689878 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689887 5139 cpu_manager.go:414] "containerMap: removing container" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689891 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689900 5139 cpu_manager.go:414] "containerMap: removing container" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689905 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689917 5139 cpu_manager.go:414] "containerMap: removing container" podUID="1dbb1410-3b3b-4a28-bb15-92efac37aa44" containerName="main" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689921 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbb1410-3b3b-4a28-bb15-92efac37aa44" containerName="main" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689932 5139 cpu_manager.go:414] "containerMap: removing container" podUID="17c8ec1b-452e-49b5-8708-4093b854ef9b" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689937 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17c8ec1b-452e-49b5-8708-4093b854ef9b" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689947 5139 cpu_manager.go:414] "containerMap: removing container" podUID="baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689952 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689963 5139 cpu_manager.go:414] "containerMap: removing container" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689967 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689974 5139 cpu_manager.go:414] "containerMap: removing container" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689978 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689985 5139 cpu_manager.go:414] "containerMap: removing container" podUID="09bccc97-05ea-4d94-8887-c0d6f1e3bd19" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.689989 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bccc97-05ea-4d94-8887-c0d6f1e3bd19" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690050 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690057 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc58be14-16c1-4d36-a08e-9736d0c862dd" containerName="istio-proxy" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690063 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690071 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbb1410-3b3b-4a28-bb15-92efac37aa44" containerName="main" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690077 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="17c8ec1b-452e-49b5-8708-4093b854ef9b" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690083 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="baa29a9a-d6e4-4c5c-86f7-bf6bed55faf4" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690089 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="772c18d4-c0b2-4aa7-952d-25e11e4a945f" containerName="maas-api" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690094 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="90a132e5-9eef-4f67-a14f-5d74e05f4e75" containerName="main" Oct 30 22:50:23.690092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.690099 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bccc97-05ea-4d94-8887-c0d6f1e3bd19" containerName="maas-api" Oct 30 22:50:23.694577 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.694553 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.743653 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.743618 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-9af9e7-openshift-default-dockercfg-568fh\"" Oct 30 22:50:23.805916 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.805884 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl"] Oct 30 22:50:23.823527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823493 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-credential-socket\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.823527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823527 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istiod-ca-cert\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.823791 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823655 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-data\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.823791 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823681 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-token\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.823926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823790 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-certs\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.823926 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823850 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527hx\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-kube-api-access-527hx\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.824045 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823949 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-envoy\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.824045 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823972 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-podinfo\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.824045 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.823995 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-socket\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.925583 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.925545 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-certs\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.925773 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.925592 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527hx\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-kube-api-access-527hx\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.925773 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.925719 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-envoy\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.925773 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.925763 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-podinfo\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.925900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.925804 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-socket\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.925900 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.925889 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-credential-socket\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.926008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.925915 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istiod-ca-cert\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.926067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.926015 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-data\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.926067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.926024 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-certs\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.926067 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.926041 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-token\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.926261 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.926233 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-socket\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.926492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.926463 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-credential-socket\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.926581 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.926502 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-data\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.926744 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.926724 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istiod-ca-cert\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.928085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.928056 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-envoy\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.928312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.928295 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-podinfo\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.984833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.984748 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-token\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:23.987950 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:23.987924 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-527hx\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-kube-api-access-527hx\") pod \"e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:24.008070 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:24.008035 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:24.432043 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:24.432008 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl"] Oct 30 22:50:24.439547 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:50:24.439515 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1c4ca3_b3cb_4b0b_8456_4b3eca8aafa3.slice/crio-2214e4e716a08d29ad291d9da122008ef584e1730eb5347e90cacd18858513ef WatchSource:0}: Error finding container 2214e4e716a08d29ad291d9da122008ef584e1730eb5347e90cacd18858513ef: Status 404 returned error can't find the container with id 2214e4e716a08d29ad291d9da122008ef584e1730eb5347e90cacd18858513ef Oct 30 22:50:24.442092 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:24.442055 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:50:24.442191 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:24.442123 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:50:24.442191 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:24.442154 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:50:24.958274 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:24.958236 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" event={"ID":"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3","Type":"ContainerStarted","Data":"6396bfd24a36d188da1c7106b61d04a242ffa2ad9daf0edfc90ec7ff6adc6d39"} Oct 30 22:50:24.958692 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:24.958298 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" event={"ID":"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3","Type":"ContainerStarted","Data":"2214e4e716a08d29ad291d9da122008ef584e1730eb5347e90cacd18858513ef"} Oct 30 22:50:25.008246 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:25.008208 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:25.014272 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:25.014246 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:25.074183 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:25.074135 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" podStartSLOduration=2.074122412 podStartE2EDuration="2.074122412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:25.07207663 +0000 UTC m=+4333.663362116" watchObservedRunningTime="2026-10-30 22:50:25.074122412 +0000 UTC m=+4333.665407899" Oct 30 22:50:25.963769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:25.963734 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:25.964735 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:25.964709 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:50:26.570928 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:26.570892 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" podUID="4a0e33b4-d4e4-468f-b9b4-c78654a41f13" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:50:31.650306 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.650268 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9"] Oct 30 22:50:31.654299 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.654272 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:31.750536 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.750506 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9af9e7\"" Oct 30 22:50:31.750536 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.750508 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9af9e7-dockercfg-r7wcj\"" Oct 30 22:50:31.752605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.752583 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9"] Oct 30 22:50:31.815133 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.815096 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9\" (UID: \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:31.815324 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.815174 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnjdb\" (UniqueName: \"kubernetes.io/projected/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-kube-api-access-lnjdb\") pod \"payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9\" (UID: \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:31.916343 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.916251 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9\" (UID: \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:31.916527 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.916352 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnjdb\" (UniqueName: \"kubernetes.io/projected/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-kube-api-access-lnjdb\") pod \"payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9\" (UID: \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:31.917046 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.917025 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9\" (UID: \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:31.974777 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:31.974740 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnjdb\" (UniqueName: \"kubernetes.io/projected/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-kube-api-access-lnjdb\") pod \"payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9\" (UID: \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:32.267266 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:32.267230 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:32.651488 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:32.651397 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9"] Oct 30 22:50:32.659689 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:50:32.659656 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc162e80a_4ff7_49f6_8d7c_7de3b6f34877.slice/crio-b58d2839dc5de14f60ac538e54874c479a4231552e1b67b6b6766aab14c4339c WatchSource:0}: Error finding container b58d2839dc5de14f60ac538e54874c479a4231552e1b67b6b6766aab14c4339c: Status 404 returned error can't find the container with id b58d2839dc5de14f60ac538e54874c479a4231552e1b67b6b6766aab14c4339c Oct 30 22:50:33.004562 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.004525 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" event={"ID":"c162e80a-4ff7-49f6-8d7c-7de3b6f34877","Type":"ContainerStarted","Data":"d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd"} Oct 30 22:50:33.004743 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.004585 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" event={"ID":"c162e80a-4ff7-49f6-8d7c-7de3b6f34877","Type":"ContainerStarted","Data":"b58d2839dc5de14f60ac538e54874c479a4231552e1b67b6b6766aab14c4339c"} Oct 30 22:50:33.004743 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.004736 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:33.115753 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.115699 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" podStartSLOduration=2.115687011 podStartE2EDuration="2.115687011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:33.11412672 +0000 UTC m=+4341.705412206" watchObservedRunningTime="2026-10-30 22:50:33.115687011 +0000 UTC m=+4341.706972513" Oct 30 22:50:33.205543 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.205479 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:50:33.331517 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.331487 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-plugins-config-volume\") pod \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\" (UID: \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\") " Oct 30 22:50:33.331704 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.331687 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-kube-api-access-rlxkn\" (UniqueName: \"kubernetes.io/projected/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-kube-api-access-rlxkn\") pod \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\" (UID: \"4a0e33b4-d4e4-468f-b9b4-c78654a41f13\") " Oct 30 22:50:33.331921 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.331895 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-plugins-config-volume" pod "4a0e33b4-d4e4-468f-b9b4-c78654a41f13" (UID: "4a0e33b4-d4e4-468f-b9b4-c78654a41f13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:50:33.333855 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.333832 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-kube-api-access-rlxkn" pod "4a0e33b4-d4e4-468f-b9b4-c78654a41f13" (UID: "4a0e33b4-d4e4-468f-b9b4-c78654a41f13"). InnerVolumeSpecName "kube-api-access-rlxkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:50:33.432955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.432923 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlxkn\" (UniqueName: \"kubernetes.io/projected/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-kube-api-access-rlxkn\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:33.432955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:33.432954 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a0e33b4-d4e4-468f-b9b4-c78654a41f13-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:50:34.012603 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.012568 5139 generic.go:356] "Generic (PLEG): container finished" podID="4a0e33b4-d4e4-468f-b9b4-c78654a41f13" containerID="6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a" exitCode=0 Oct 30 22:50:34.013012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.012629 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" Oct 30 22:50:34.013012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.012650 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" event={"ID":"4a0e33b4-d4e4-468f-b9b4-c78654a41f13","Type":"ContainerDied","Data":"6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a"} Oct 30 22:50:34.013012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.012763 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng" event={"ID":"4a0e33b4-d4e4-468f-b9b4-c78654a41f13","Type":"ContainerDied","Data":"a44e4772f2f88225a7546e4582be620bef033c9b3e91b7b26c99f83ae6022153"} Oct 30 22:50:34.013012 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.012797 5139 scope.go:122] "RemoveContainer" containerID="6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a" Oct 30 22:50:34.022501 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.022480 5139 scope.go:122] "RemoveContainer" containerID="6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a" Oct 30 22:50:34.022841 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:50:34.022806 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a\": container with ID starting with 6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a not found: ID does not exist" containerID="6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a" Oct 30 22:50:34.022971 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.022853 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a"} err="failed to get container status \"6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a\": rpc error: code = NotFound desc = could not find container \"6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a\": container with ID starting with 6242e7ee4ff3d086979d85735eecad1cc001d8e516e50b89d38a7541e8fd8c2a not found: ID does not exist" Oct 30 22:50:34.166673 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.166640 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng"] Oct 30 22:50:34.213050 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:34.213019 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-5e884c-5b47bc5587-l7cng"] Oct 30 22:50:35.541721 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:35.541689 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a0e33b4-d4e4-468f-b9b4-c78654a41f13" path="/var/lib/kubelet/pods/4a0e33b4-d4e4-468f-b9b4-c78654a41f13/volumes" Oct 30 22:50:44.015994 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:50:44.015945 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:50:49.804508 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:50:49.804473 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:51:10.525866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:10.525830 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w"] Oct 30 22:51:10.526442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:10.526060 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" containerID="cri-o://72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d" gracePeriod=30 Oct 30 22:51:15.778488 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.778462 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 22:51:15.796264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796209 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-credential-socket\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796264 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796253 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-certs\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796278 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-data\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796339 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-podinfo\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796363 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-kube-api-access-956ct\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-kube-api-access-956ct\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796451 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796439 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-socket\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796680 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796487 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-envoy\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796680 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796513 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-credential-socket" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:15.796680 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796584 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-token\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-token\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796680 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796610 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-certs" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:15.796680 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796650 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istiod-ca-cert\") pod \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\" (UID: \"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef\") " Oct 30 22:51:15.796959 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796731 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-socket" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:15.797021 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.796957 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-data" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:15.797072 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.797040 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istiod-ca-cert" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:15.797128 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.797110 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.797174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.797134 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.797174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.797148 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.797174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.797161 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.798663 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.798637 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-podinfo" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:51:15.798846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.798826 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-token" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:15.799000 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.798981 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-kube-api-access-956ct" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "kube-api-access-956ct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:15.799090 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.799068 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-envoy" pod "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" (UID: "c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:51:15.898492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.898462 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.898492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.898489 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-956ct\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-kube-api-access-956ct\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.898492 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.898500 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.898722 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.898512 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:15.898722 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:15.898520 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:16.242818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.242783 5139 generic.go:356] "Generic (PLEG): container finished" podID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerID="72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d" exitCode=0 Oct 30 22:51:16.242992 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.242835 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" Oct 30 22:51:16.242992 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.242861 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" event={"ID":"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef","Type":"ContainerDied","Data":"72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d"} Oct 30 22:51:16.242992 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.242919 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w" event={"ID":"c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef","Type":"ContainerDied","Data":"c28eaabe58434297d7bd1be3387b86b4c65d042f2243dc901e3c6a52bdc483fa"} Oct 30 22:51:16.242992 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.242949 5139 scope.go:122] "RemoveContainer" containerID="72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d" Oct 30 22:51:16.253295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.253277 5139 scope.go:122] "RemoveContainer" containerID="7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88" Oct 30 22:51:16.261683 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.261660 5139 scope.go:122] "RemoveContainer" containerID="72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d" Oct 30 22:51:16.261925 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:51:16.261907 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d\": container with ID starting with 72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d not found: ID does not exist" containerID="72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d" Oct 30 22:51:16.261976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.261935 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d"} err="failed to get container status \"72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d\": rpc error: code = NotFound desc = could not find container \"72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d\": container with ID starting with 72bf57af4bf377073868e4737861105d32daff918411c16f7cdb75b5954f938d not found: ID does not exist" Oct 30 22:51:16.261976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.261953 5139 scope.go:122] "RemoveContainer" containerID="7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88" Oct 30 22:51:16.262180 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:51:16.262165 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88\": container with ID starting with 7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88 not found: ID does not exist" containerID="7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88" Oct 30 22:51:16.262226 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.262186 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88"} err="failed to get container status \"7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88\": rpc error: code = NotFound desc = could not find container \"7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88\": container with ID starting with 7bc163763eac2374327224f39b89819bf05f0f7ac6f0213734f07f9536e0fe88 not found: ID does not exist" Oct 30 22:51:16.419447 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.419402 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w"] Oct 30 22:51:16.465934 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:16.465907 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-4pq5w"] Oct 30 22:51:17.542166 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:17.542133 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" path="/var/lib/kubelet/pods/c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef/volumes" Oct 30 22:51:19.803940 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:51:19.803886 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:51:27.384872 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.384829 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq"] Oct 30 22:51:27.387491 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.385090 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" podUID="4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" containerName="maas-api" containerID="cri-o://51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760" gracePeriod=30 Oct 30 22:51:27.635813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.635760 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:51:27.724641 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.724606 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls\") pod \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " Oct 30 22:51:27.724814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.724796 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-kube-api-access-jgvjf\" (UniqueName: \"kubernetes.io/projected/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-kube-api-access-jgvjf\") pod \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " Oct 30 22:51:27.724867 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.724852 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-metrics-certs\") pod \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " Oct 30 22:51:27.724937 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.724921 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-openshift-service-ca\") pod \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\" (UID: \"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc\") " Oct 30 22:51:27.725359 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.725321 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-openshift-service-ca" pod "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" (UID: "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:51:27.726998 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.726970 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-kube-api-access-jgvjf" pod "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" (UID: "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc"). InnerVolumeSpecName "kube-api-access-jgvjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:51:27.727210 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.727181 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-metrics-certs" pod "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" (UID: "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:27.727314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.727215 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls" pod "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" (UID: "4c240dd4-fd65-4f9d-89d6-593d39ae0ebc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:51:27.827036 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.827001 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:27.827036 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.827031 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:27.827036 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.827041 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgvjf\" (UniqueName: \"kubernetes.io/projected/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-kube-api-access-jgvjf\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:27.827341 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:27.827053 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:51:28.304718 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.304680 5139 generic.go:356] "Generic (PLEG): container finished" podID="4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" containerID="51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760" exitCode=0 Oct 30 22:51:28.304905 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.304730 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" Oct 30 22:51:28.304905 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.304737 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" event={"ID":"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc","Type":"ContainerDied","Data":"51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760"} Oct 30 22:51:28.304905 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.304801 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq" event={"ID":"4c240dd4-fd65-4f9d-89d6-593d39ae0ebc","Type":"ContainerDied","Data":"ebb07d4fab3e22d0efdfaf068611b31bcf16a4a88459ce78d61409be1dafadfe"} Oct 30 22:51:28.304905 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.304832 5139 scope.go:122] "RemoveContainer" containerID="51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760" Oct 30 22:51:28.315317 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.315268 5139 scope.go:122] "RemoveContainer" containerID="51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760" Oct 30 22:51:28.315890 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:51:28.315861 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760\": container with ID starting with 51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760 not found: ID does not exist" containerID="51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760" Oct 30 22:51:28.316008 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.315902 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760"} err="failed to get container status \"51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760\": rpc error: code = NotFound desc = could not find container \"51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760\": container with ID starting with 51cd1f7180f9c7dcee38bdae740723a8eae06e90cf615760dacef1dc43c0c760 not found: ID does not exist" Oct 30 22:51:28.460204 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.460169 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq"] Oct 30 22:51:28.465600 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:28.465575 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-ae7a1f-7656897587-hjrjq"] Oct 30 22:51:29.541973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:29.541942 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" path="/var/lib/kubelet/pods/4c240dd4-fd65-4f9d-89d6-593d39ae0ebc/volumes" Oct 30 22:51:40.372611 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:40.372576 5139 generic.go:356] "Generic (PLEG): container finished" podID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerID="6396bfd24a36d188da1c7106b61d04a242ffa2ad9daf0edfc90ec7ff6adc6d39" exitCode=0 Oct 30 22:51:40.373053 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:40.372645 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" event={"ID":"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3","Type":"ContainerDied","Data":"6396bfd24a36d188da1c7106b61d04a242ffa2ad9daf0edfc90ec7ff6adc6d39"} Oct 30 22:51:40.373053 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:40.373043 5139 scope.go:122] "RemoveContainer" containerID="6396bfd24a36d188da1c7106b61d04a242ffa2ad9daf0edfc90ec7ff6adc6d39" Oct 30 22:51:40.374551 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:40.374524 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:51:40.374634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:40.374590 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:51:40.374634 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:40.374620 5139 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"} Oct 30 22:51:40.964567 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:40.964533 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:51:41.391763 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:41.391730 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" event={"ID":"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3","Type":"ContainerStarted","Data":"073904ab591c812ca625ac41c950c8f0200459948bac439085a947e912cf7c3a"} Oct 30 22:51:42.008408 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:42.008378 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:51:42.009614 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:42.009594 5139 patch_prober.go:29] interesting pod/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:51:42.009676 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:42.009652 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 22:51:43.008809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:43.008770 5139 patch_prober.go:29] interesting pod/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:51:43.009209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:43.008838 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 22:51:44.009196 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:44.009159 5139 patch_prober.go:29] interesting pod/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:51:44.009605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:44.009214 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 22:51:45.008539 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:45.008503 5139 patch_prober.go:29] interesting pod/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:51:45.008729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:45.008557 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 22:51:45.257551 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:45.257514 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht"] Oct 30 22:51:45.303951 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:45.303875 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv"] Oct 30 22:51:46.008740 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:46.008708 5139 patch_prober.go:29] interesting pod/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 22:51:46.008944 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:46.008764 5139 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 22:51:47.013773 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:47.013741 5139 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:51:47.014183 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:47.014006 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:51:47.014694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:47.014674 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:51:49.804553 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:51:49.804516 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:51:52.376781 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:52.376692 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh"] Oct 30 22:51:57.377495 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:51:57.377433 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" podUID="690d9602-55a9-41f5-8bf0-506dd9e9bcae" containerName="payload-pre-processing" containerID="cri-o://39f3b14477c4182b39954121830639fd0ab6afe68e14f0f065828d54a6449237" gracePeriod=25 Oct 30 22:52:00.302128 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.302085 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" containerID="cri-o://4cb7ff5d69dec4b604abf0017d7b1f120fd3f166523405d8566b5daff24eabc7" gracePeriod=45 Oct 30 22:52:00.327832 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.327802 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" containerID="cri-o://df55013654d75b42d1add9ce5b931f01714e6c15a71d7d965163c0f979537575" gracePeriod=45 Oct 30 22:52:00.396726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.396692 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Oct 30 22:52:00.489695 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.489655 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused" Oct 30 22:52:00.489869 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.489778 5139 generic.go:356] "Generic (PLEG): container finished" podID="aba9092c-a737-48f9-99c4-675c8d0da106" containerID="4cb7ff5d69dec4b604abf0017d7b1f120fd3f166523405d8566b5daff24eabc7" exitCode=0 Oct 30 22:52:00.489956 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.489898 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" event={"ID":"aba9092c-a737-48f9-99c4-675c8d0da106","Type":"ContainerDied","Data":"4cb7ff5d69dec4b604abf0017d7b1f120fd3f166523405d8566b5daff24eabc7"} Oct 30 22:52:00.491661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.491640 5139 generic.go:356] "Generic (PLEG): container finished" podID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerID="df55013654d75b42d1add9ce5b931f01714e6c15a71d7d965163c0f979537575" exitCode=0 Oct 30 22:52:00.491750 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.491710 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" event={"ID":"9249409a-b8c4-4b9b-908a-2402b39d8f4a","Type":"ContainerDied","Data":"df55013654d75b42d1add9ce5b931f01714e6c15a71d7d965163c0f979537575"} Oct 30 22:52:00.599356 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.599333 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 22:52:00.604632 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.604610 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 22:52:00.700327 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700299 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-home\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-home\") pod \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " Oct 30 22:52:00.700526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700335 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-dshm\") pod \"aba9092c-a737-48f9-99c4-675c8d0da106\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " Oct 30 22:52:00.700526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700376 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-model-cache\") pod \"aba9092c-a737-48f9-99c4-675c8d0da106\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " Oct 30 22:52:00.700526 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700404 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-home\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-home\") pod \"aba9092c-a737-48f9-99c4-675c8d0da106\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " Oct 30 22:52:00.700697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700533 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-dshm\") pod \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " Oct 30 22:52:00.700697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700600 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-home" pod "9249409a-b8c4-4b9b-908a-2402b39d8f4a" (UID: "9249409a-b8c4-4b9b-908a-2402b39d8f4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:00.700822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700713 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-model-cache" pod "aba9092c-a737-48f9-99c4-675c8d0da106" (UID: "aba9092c-a737-48f9-99c4-675c8d0da106"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:00.700822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700738 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-home" pod "aba9092c-a737-48f9-99c4-675c8d0da106" (UID: "aba9092c-a737-48f9-99c4-675c8d0da106"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:00.700930 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700830 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-tmp-dir\") pod \"aba9092c-a737-48f9-99c4-675c8d0da106\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " Oct 30 22:52:00.700930 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700881 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tmp-dir\") pod \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " Oct 30 22:52:00.700930 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700904 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-tmp-dir" pod "aba9092c-a737-48f9-99c4-675c8d0da106" (UID: "aba9092c-a737-48f9-99c4-675c8d0da106"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:00.700930 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700913 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9249409a-b8c4-4b9b-908a-2402b39d8f4a-kube-api-access-msh99\" (UniqueName: \"kubernetes.io/projected/9249409a-b8c4-4b9b-908a-2402b39d8f4a-kube-api-access-msh99\") pod \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " Oct 30 22:52:00.701136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.700985 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tls-certs\") pod \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " Oct 30 22:52:00.701136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701031 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba9092c-a737-48f9-99c4-675c8d0da106-kube-api-access-dvxxr\" (UniqueName: \"kubernetes.io/projected/aba9092c-a737-48f9-99c4-675c8d0da106-kube-api-access-dvxxr\") pod \"aba9092c-a737-48f9-99c4-675c8d0da106\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " Oct 30 22:52:00.701136 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701069 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-model-cache\") pod \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\" (UID: \"9249409a-b8c4-4b9b-908a-2402b39d8f4a\") " Oct 30 22:52:00.701286 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701134 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tmp-dir" pod "9249409a-b8c4-4b9b-908a-2402b39d8f4a" (UID: "9249409a-b8c4-4b9b-908a-2402b39d8f4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:00.701286 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701154 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aba9092c-a737-48f9-99c4-675c8d0da106-tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9092c-a737-48f9-99c4-675c8d0da106-tls-certs\") pod \"aba9092c-a737-48f9-99c4-675c8d0da106\" (UID: \"aba9092c-a737-48f9-99c4-675c8d0da106\") " Oct 30 22:52:00.701588 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701570 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.701745 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701731 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.701851 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701838 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.701987 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701975 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.702105 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.702092 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.702244 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.701951 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-model-cache" pod "9249409a-b8c4-4b9b-908a-2402b39d8f4a" (UID: "9249409a-b8c4-4b9b-908a-2402b39d8f4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:00.703239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.703217 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-dshm" pod "9249409a-b8c4-4b9b-908a-2402b39d8f4a" (UID: "9249409a-b8c4-4b9b-908a-2402b39d8f4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:00.703820 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.703782 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-dshm" pod "aba9092c-a737-48f9-99c4-675c8d0da106" (UID: "aba9092c-a737-48f9-99c4-675c8d0da106"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:00.704203 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.704167 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba9092c-a737-48f9-99c4-675c8d0da106-kube-api-access-dvxxr" pod "aba9092c-a737-48f9-99c4-675c8d0da106" (UID: "aba9092c-a737-48f9-99c4-675c8d0da106"). InnerVolumeSpecName "kube-api-access-dvxxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:00.704605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.704565 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aba9092c-a737-48f9-99c4-675c8d0da106-tls-certs" pod "aba9092c-a737-48f9-99c4-675c8d0da106" (UID: "aba9092c-a737-48f9-99c4-675c8d0da106"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:00.704702 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.704659 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tls-certs" pod "9249409a-b8c4-4b9b-908a-2402b39d8f4a" (UID: "9249409a-b8c4-4b9b-908a-2402b39d8f4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:00.704811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.704793 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9249409a-b8c4-4b9b-908a-2402b39d8f4a-kube-api-access-msh99" pod "9249409a-b8c4-4b9b-908a-2402b39d8f4a" (UID: "9249409a-b8c4-4b9b-908a-2402b39d8f4a"). InnerVolumeSpecName "kube-api-access-msh99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:00.803441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.803379 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvxxr\" (UniqueName: \"kubernetes.io/projected/aba9092c-a737-48f9-99c4-675c8d0da106-kube-api-access-dvxxr\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.803441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.803409 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.803441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.803441 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aba9092c-a737-48f9-99c4-675c8d0da106-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.803441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.803450 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aba9092c-a737-48f9-99c4-675c8d0da106-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.803809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.803459 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9249409a-b8c4-4b9b-908a-2402b39d8f4a-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.803809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.803467 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msh99\" (UniqueName: \"kubernetes.io/projected/9249409a-b8c4-4b9b-908a-2402b39d8f4a-kube-api-access-msh99\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:00.803809 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:00.803475 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9249409a-b8c4-4b9b-908a-2402b39d8f4a-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:01.498580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.498549 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" Oct 30 22:52:01.498580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.498570 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht" event={"ID":"aba9092c-a737-48f9-99c4-675c8d0da106","Type":"ContainerDied","Data":"5b0547d36bec79315b53f855562fef7bf080e864162e19ec70c6e02d10994cf2"} Oct 30 22:52:01.499105 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.498621 5139 scope.go:122] "RemoveContainer" containerID="4cb7ff5d69dec4b604abf0017d7b1f120fd3f166523405d8566b5daff24eabc7" Oct 30 22:52:01.500243 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.500004 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" Oct 30 22:52:01.500243 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.500002 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv" event={"ID":"9249409a-b8c4-4b9b-908a-2402b39d8f4a","Type":"ContainerDied","Data":"1bfb28a54249489db1949945b4df055e87859c7ccb9db490fb05acd7129ed327"} Oct 30 22:52:01.508276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.508259 5139 scope.go:122] "RemoveContainer" containerID="df55013654d75b42d1add9ce5b931f01714e6c15a71d7d965163c0f979537575" Oct 30 22:52:01.678991 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.678959 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht"] Oct 30 22:52:01.725401 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.725367 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/premium-w3-4d14e0-kserve-7c5b98d799-qdhht"] Oct 30 22:52:01.801051 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.800962 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv"] Oct 30 22:52:01.811113 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:01.811084 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/simulator-w3-4d14e0-kserve-f4c67f658-grdzv"] Oct 30 22:52:03.542496 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:03.542464 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" path="/var/lib/kubelet/pods/9249409a-b8c4-4b9b-908a-2402b39d8f4a/volumes" Oct 30 22:52:03.542955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:03.542940 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" path="/var/lib/kubelet/pods/aba9092c-a737-48f9-99c4-675c8d0da106/volumes" Oct 30 22:52:04.747597 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:04.747555 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" podUID="690d9602-55a9-41f5-8bf0-506dd9e9bcae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:52:08.641754 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:08.641715 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc"] Oct 30 22:52:08.687942 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:08.687912 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n"] Oct 30 22:52:12.764164 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:12.764135 5139 scope.go:122] "RemoveContainer" containerID="958dba0cfaaeba443a3420772f1c79fd738e523afd1ec350e84ac0be8705e72b" Oct 30 22:52:14.746366 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:14.746320 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" podUID="690d9602-55a9-41f5-8bf0-506dd9e9bcae" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:52:17.588369 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.588323 5139 generic.go:356] "Generic (PLEG): container finished" podID="690d9602-55a9-41f5-8bf0-506dd9e9bcae" containerID="39f3b14477c4182b39954121830639fd0ab6afe68e14f0f065828d54a6449237" exitCode=0 Oct 30 22:52:17.588846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.588403 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" event={"ID":"690d9602-55a9-41f5-8bf0-506dd9e9bcae","Type":"ContainerDied","Data":"39f3b14477c4182b39954121830639fd0ab6afe68e14f0f065828d54a6449237"} Oct 30 22:52:17.635443 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.635378 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 22:52:17.674209 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.674184 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/690d9602-55a9-41f5-8bf0-506dd9e9bcae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690d9602-55a9-41f5-8bf0-506dd9e9bcae-plugins-config-volume\") pod \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\" (UID: \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\") " Oct 30 22:52:17.674318 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.674224 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690d9602-55a9-41f5-8bf0-506dd9e9bcae-kube-api-access-bd2gh\" (UniqueName: \"kubernetes.io/projected/690d9602-55a9-41f5-8bf0-506dd9e9bcae-kube-api-access-bd2gh\") pod \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\" (UID: \"690d9602-55a9-41f5-8bf0-506dd9e9bcae\") " Oct 30 22:52:17.674620 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.674599 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/690d9602-55a9-41f5-8bf0-506dd9e9bcae-plugins-config-volume" pod "690d9602-55a9-41f5-8bf0-506dd9e9bcae" (UID: "690d9602-55a9-41f5-8bf0-506dd9e9bcae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:52:17.676248 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.676230 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690d9602-55a9-41f5-8bf0-506dd9e9bcae-kube-api-access-bd2gh" pod "690d9602-55a9-41f5-8bf0-506dd9e9bcae" (UID: "690d9602-55a9-41f5-8bf0-506dd9e9bcae"). InnerVolumeSpecName "kube-api-access-bd2gh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:17.776063 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.776024 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/690d9602-55a9-41f5-8bf0-506dd9e9bcae-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:17.776063 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:17.776064 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd2gh\" (UniqueName: \"kubernetes.io/projected/690d9602-55a9-41f5-8bf0-506dd9e9bcae-kube-api-access-bd2gh\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:18.594726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:18.594695 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" Oct 30 22:52:18.594726 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:18.594712 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh" event={"ID":"690d9602-55a9-41f5-8bf0-506dd9e9bcae","Type":"ContainerDied","Data":"ff712d31d16acb1e77a7b9f9d00223e4233d1560e72731998856f0ac8ef3ae3b"} Oct 30 22:52:18.595276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:18.594788 5139 scope.go:122] "RemoveContainer" containerID="39f3b14477c4182b39954121830639fd0ab6afe68e14f0f065828d54a6449237" Oct 30 22:52:18.748239 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:18.748203 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh"] Oct 30 22:52:18.794610 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:18.794579 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-4d14e0-5c95b4c4b7-lrtgh"] Oct 30 22:52:19.542544 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:19.542505 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690d9602-55a9-41f5-8bf0-506dd9e9bcae" path="/var/lib/kubelet/pods/690d9602-55a9-41f5-8bf0-506dd9e9bcae/volumes" Oct 30 22:52:19.804359 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:52:19.804276 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:52:23.661670 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:23.661617 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" containerID="cri-o://3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9" gracePeriod=45 Oct 30 22:52:23.707285 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:23.707249 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" containerID="cri-o://1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd" gracePeriod=45 Oct 30 22:52:23.774976 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:23.774944 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": dial tcp 10.134.0.63:8000: connect: connection refused" Oct 30 22:52:23.968729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:23.968698 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 22:52:23.993244 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:23.993222 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 22:52:24.043959 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.043935 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-home\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-home\") pod \"a8a27a42-dfc1-491d-b955-774c21708323\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " Oct 30 22:52:24.044091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.043968 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a27a42-dfc1-491d-b955-774c21708323-kube-api-access-dsmkf\" (UniqueName: \"kubernetes.io/projected/a8a27a42-dfc1-491d-b955-774c21708323-kube-api-access-dsmkf\") pod \"a8a27a42-dfc1-491d-b955-774c21708323\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " Oct 30 22:52:24.044091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044006 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-dshm\") pod \"a8a27a42-dfc1-491d-b955-774c21708323\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " Oct 30 22:52:24.044091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044035 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-dshm\") pod \"a377639d-cd1a-4182-8067-fb2071641be1\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " Oct 30 22:52:24.044091 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044066 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-home\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-home\") pod \"a377639d-cd1a-4182-8067-fb2071641be1\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " Oct 30 22:52:24.044314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044123 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-model-cache\") pod \"a8a27a42-dfc1-491d-b955-774c21708323\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " Oct 30 22:52:24.044314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044196 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a27a42-dfc1-491d-b955-774c21708323-tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a27a42-dfc1-491d-b955-774c21708323-tls-certs\") pod \"a8a27a42-dfc1-491d-b955-774c21708323\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " Oct 30 22:52:24.044314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044232 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a377639d-cd1a-4182-8067-fb2071641be1-tls-certs\" (UniqueName: \"kubernetes.io/secret/a377639d-cd1a-4182-8067-fb2071641be1-tls-certs\") pod \"a377639d-cd1a-4182-8067-fb2071641be1\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " Oct 30 22:52:24.044314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044246 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-home" pod "a8a27a42-dfc1-491d-b955-774c21708323" (UID: "a8a27a42-dfc1-491d-b955-774c21708323"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.044314 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044285 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-model-cache\") pod \"a377639d-cd1a-4182-8067-fb2071641be1\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " Oct 30 22:52:24.044626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044332 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-tmp-dir\") pod \"a377639d-cd1a-4182-8067-fb2071641be1\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " Oct 30 22:52:24.044626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044369 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-tmp-dir\") pod \"a8a27a42-dfc1-491d-b955-774c21708323\" (UID: \"a8a27a42-dfc1-491d-b955-774c21708323\") " Oct 30 22:52:24.044626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044376 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-home" pod "a377639d-cd1a-4182-8067-fb2071641be1" (UID: "a377639d-cd1a-4182-8067-fb2071641be1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.044626 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044459 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a377639d-cd1a-4182-8067-fb2071641be1-kube-api-access-5lhrg\" (UniqueName: \"kubernetes.io/projected/a377639d-cd1a-4182-8067-fb2071641be1-kube-api-access-5lhrg\") pod \"a377639d-cd1a-4182-8067-fb2071641be1\" (UID: \"a377639d-cd1a-4182-8067-fb2071641be1\") " Oct 30 22:52:24.044866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044687 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-model-cache" pod "a8a27a42-dfc1-491d-b955-774c21708323" (UID: "a8a27a42-dfc1-491d-b955-774c21708323"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.044866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044772 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-tmp-dir" pod "a377639d-cd1a-4182-8067-fb2071641be1" (UID: "a377639d-cd1a-4182-8067-fb2071641be1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.045040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.044634 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-model-cache" pod "a377639d-cd1a-4182-8067-fb2071641be1" (UID: "a377639d-cd1a-4182-8067-fb2071641be1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.045040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.045021 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-tmp-dir" pod "a8a27a42-dfc1-491d-b955-774c21708323" (UID: "a8a27a42-dfc1-491d-b955-774c21708323"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.045040 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.045037 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.045208 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.045054 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.045208 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.045068 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.045208 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.045084 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.046735 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.046710 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a377639d-cd1a-4182-8067-fb2071641be1-tls-certs" pod "a377639d-cd1a-4182-8067-fb2071641be1" (UID: "a377639d-cd1a-4182-8067-fb2071641be1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:24.046852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.046755 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-dshm" pod "a8a27a42-dfc1-491d-b955-774c21708323" (UID: "a8a27a42-dfc1-491d-b955-774c21708323"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.046852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.046812 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-dshm" pod "a377639d-cd1a-4182-8067-fb2071641be1" (UID: "a377639d-cd1a-4182-8067-fb2071641be1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:52:24.046852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.046841 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a27a42-dfc1-491d-b955-774c21708323-kube-api-access-dsmkf" pod "a8a27a42-dfc1-491d-b955-774c21708323" (UID: "a8a27a42-dfc1-491d-b955-774c21708323"). InnerVolumeSpecName "kube-api-access-dsmkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:24.046852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.046842 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a27a42-dfc1-491d-b955-774c21708323-tls-certs" pod "a8a27a42-dfc1-491d-b955-774c21708323" (UID: "a8a27a42-dfc1-491d-b955-774c21708323"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:52:24.047063 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.047055 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a377639d-cd1a-4182-8067-fb2071641be1-kube-api-access-5lhrg" pod "a377639d-cd1a-4182-8067-fb2071641be1" (UID: "a377639d-cd1a-4182-8067-fb2071641be1"). InnerVolumeSpecName "kube-api-access-5lhrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:52:24.146044 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.146006 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsmkf\" (UniqueName: \"kubernetes.io/projected/a8a27a42-dfc1-491d-b955-774c21708323-kube-api-access-dsmkf\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.146044 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.146039 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.146044 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.146053 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.146295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.146065 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a8a27a42-dfc1-491d-b955-774c21708323-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.146295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.146078 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a377639d-cd1a-4182-8067-fb2071641be1-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.146295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.146089 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a377639d-cd1a-4182-8067-fb2071641be1-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.146295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.146102 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a8a27a42-dfc1-491d-b955-774c21708323-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.146295 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.146114 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lhrg\" (UniqueName: \"kubernetes.io/projected/a377639d-cd1a-4182-8067-fb2071641be1-kube-api-access-5lhrg\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:52:24.626114 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.626080 5139 generic.go:356] "Generic (PLEG): container finished" podID="a8a27a42-dfc1-491d-b955-774c21708323" containerID="1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd" exitCode=0 Oct 30 22:52:24.626319 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.626135 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" Oct 30 22:52:24.626319 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.626164 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" event={"ID":"a8a27a42-dfc1-491d-b955-774c21708323","Type":"ContainerDied","Data":"1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd"} Oct 30 22:52:24.626319 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.626201 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n" event={"ID":"a8a27a42-dfc1-491d-b955-774c21708323","Type":"ContainerDied","Data":"e6404fee9f4e116cbbeb77ab6f1a0564e6fa19aee17291af2939ffb4d2008a29"} Oct 30 22:52:24.626319 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.626225 5139 scope.go:122] "RemoveContainer" containerID="1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd" Oct 30 22:52:24.627671 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.627644 5139 generic.go:356] "Generic (PLEG): container finished" podID="a377639d-cd1a-4182-8067-fb2071641be1" containerID="3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9" exitCode=0 Oct 30 22:52:24.627955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.627699 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" event={"ID":"a377639d-cd1a-4182-8067-fb2071641be1","Type":"ContainerDied","Data":"3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9"} Oct 30 22:52:24.627955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.627735 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" Oct 30 22:52:24.627955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.627739 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc" event={"ID":"a377639d-cd1a-4182-8067-fb2071641be1","Type":"ContainerDied","Data":"42293e25712820b3f27668df46f640464d1cfd41e651ca829280f45e3be044fe"} Oct 30 22:52:24.636880 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.636859 5139 scope.go:122] "RemoveContainer" containerID="1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd" Oct 30 22:52:24.637193 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:52:24.637174 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd\": container with ID starting with 1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd not found: ID does not exist" containerID="1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd" Oct 30 22:52:24.637257 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.637200 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd"} err="failed to get container status \"1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd\": rpc error: code = NotFound desc = could not find container \"1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd\": container with ID starting with 1694ae9dff57be2aa317ca29a38a5d2c31c070c568b71c671e986c911d8f43cd not found: ID does not exist" Oct 30 22:52:24.637257 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.637217 5139 scope.go:122] "RemoveContainer" containerID="3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9" Oct 30 22:52:24.645190 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.645172 5139 scope.go:122] "RemoveContainer" containerID="3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9" Oct 30 22:52:24.645471 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:52:24.645450 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9\": container with ID starting with 3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9 not found: ID does not exist" containerID="3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9" Oct 30 22:52:24.645541 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.645481 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9"} err="failed to get container status \"3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9\": rpc error: code = NotFound desc = could not find container \"3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9\": container with ID starting with 3249fd29913690834115cc169a3798e0ee318667d70a132d0360d27342de66d9 not found: ID does not exist" Oct 30 22:52:24.779973 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.779937 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n"] Oct 30 22:52:24.784894 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.784869 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/simulator-w2-56f3fe-kserve-66d97b5f66-7fx4n"] Oct 30 22:52:24.890479 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.890380 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc"] Oct 30 22:52:24.897827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:24.897803 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-2-w2-56f3fe-kserve-8666984cd5-bmnzc"] Oct 30 22:52:25.542442 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:25.542398 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a377639d-cd1a-4182-8067-fb2071641be1" path="/var/lib/kubelet/pods/a377639d-cd1a-4182-8067-fb2071641be1/volumes" Oct 30 22:52:25.542852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:52:25.542838 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a27a42-dfc1-491d-b955-774c21708323" path="/var/lib/kubelet/pods/a8a27a42-dfc1-491d-b955-774c21708323/volumes" Oct 30 22:52:49.803813 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:52:49.803768 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:53:10.108354 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.108319 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss"] Oct 30 22:53:10.108827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.108556 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" podUID="f60eb6af-5bbd-4a80-b835-60edc3466d68" containerName="maas-api" containerID="cri-o://bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b" gracePeriod=30 Oct 30 22:53:10.369500 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.369433 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 22:53:10.526267 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.526237 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls\") pod \"f60eb6af-5bbd-4a80-b835-60edc3466d68\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " Oct 30 22:53:10.526466 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.526287 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f60eb6af-5bbd-4a80-b835-60edc3466d68-kube-api-access-r2dtl\" (UniqueName: \"kubernetes.io/projected/f60eb6af-5bbd-4a80-b835-60edc3466d68-kube-api-access-r2dtl\") pod \"f60eb6af-5bbd-4a80-b835-60edc3466d68\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " Oct 30 22:53:10.526466 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.526311 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60eb6af-5bbd-4a80-b835-60edc3466d68-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eb6af-5bbd-4a80-b835-60edc3466d68-openshift-service-ca\") pod \"f60eb6af-5bbd-4a80-b835-60edc3466d68\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " Oct 30 22:53:10.526466 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.526430 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-metrics-certs\") pod \"f60eb6af-5bbd-4a80-b835-60edc3466d68\" (UID: \"f60eb6af-5bbd-4a80-b835-60edc3466d68\") " Oct 30 22:53:10.526790 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.526763 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60eb6af-5bbd-4a80-b835-60edc3466d68-openshift-service-ca" pod "f60eb6af-5bbd-4a80-b835-60edc3466d68" (UID: "f60eb6af-5bbd-4a80-b835-60edc3466d68"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:10.528533 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.528506 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-metrics-certs" pod "f60eb6af-5bbd-4a80-b835-60edc3466d68" (UID: "f60eb6af-5bbd-4a80-b835-60edc3466d68"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:10.528636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.528550 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls" pod "f60eb6af-5bbd-4a80-b835-60edc3466d68" (UID: "f60eb6af-5bbd-4a80-b835-60edc3466d68"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:53:10.528636 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.528578 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60eb6af-5bbd-4a80-b835-60edc3466d68-kube-api-access-r2dtl" pod "f60eb6af-5bbd-4a80-b835-60edc3466d68" (UID: "f60eb6af-5bbd-4a80-b835-60edc3466d68"). InnerVolumeSpecName "kube-api-access-r2dtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:10.627667 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.627582 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:10.627667 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.627612 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f60eb6af-5bbd-4a80-b835-60edc3466d68-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:10.627667 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.627623 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2dtl\" (UniqueName: \"kubernetes.io/projected/f60eb6af-5bbd-4a80-b835-60edc3466d68-kube-api-access-r2dtl\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:10.627667 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.627632 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eb6af-5bbd-4a80-b835-60edc3466d68-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:10.852236 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.852200 5139 generic.go:356] "Generic (PLEG): container finished" podID="f60eb6af-5bbd-4a80-b835-60edc3466d68" containerID="bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b" exitCode=0 Oct 30 22:53:10.852404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.852243 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" Oct 30 22:53:10.852404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.852283 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" event={"ID":"f60eb6af-5bbd-4a80-b835-60edc3466d68","Type":"ContainerDied","Data":"bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b"} Oct 30 22:53:10.852404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.852341 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss" event={"ID":"f60eb6af-5bbd-4a80-b835-60edc3466d68","Type":"ContainerDied","Data":"334fd158ad1876c0be41687ea5f6e3f3ff517d977e5e596192ffd3d9f1849d1c"} Oct 30 22:53:10.852404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.852364 5139 scope.go:122] "RemoveContainer" containerID="bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b" Oct 30 22:53:10.862181 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.862162 5139 scope.go:122] "RemoveContainer" containerID="bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b" Oct 30 22:53:10.862477 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:53:10.862456 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b\": container with ID starting with bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b not found: ID does not exist" containerID="bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b" Oct 30 22:53:10.862590 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:10.862483 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b"} err="failed to get container status \"bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b\": rpc error: code = NotFound desc = could not find container \"bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b\": container with ID starting with bd21f6282aa6d5ce84e6b8a2ca51950643b7e913b4e73d21afb7fe01b1ed655b not found: ID does not exist" Oct 30 22:53:11.006225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:11.006193 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss"] Oct 30 22:53:11.010835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:11.010805 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-56f3fe-c4cb5bbcd-xbjss"] Oct 30 22:53:11.541622 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:11.541591 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f60eb6af-5bbd-4a80-b835-60edc3466d68" path="/var/lib/kubelet/pods/f60eb6af-5bbd-4a80-b835-60edc3466d68/volumes" Oct 30 22:53:19.804040 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:53:19.804005 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:53:20.110754 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:20.110681 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x"] Oct 30 22:53:20.110928 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:20.110906 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" containerID="cri-o://4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad" gracePeriod=30 Oct 30 22:53:22.259076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:22.258987 5139 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 22:53:22.259076 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:22.259064 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 22:53:25.355328 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.355305 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 22:53:25.369494 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369473 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-token\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-token\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.369602 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369514 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-data\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.369644 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369607 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istiod-ca-cert\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.369644 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369632 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-envoy\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.369720 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369658 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-kube-api-access-8zqfq\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-kube-api-access-8zqfq\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.369720 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369686 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-certs\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.369720 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369712 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-credential-socket\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.369877 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369853 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-data" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:25.369939 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369879 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-socket\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.369939 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.369932 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-podinfo\") pod \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\" (UID: \"9e166eda-5b4a-4d02-9d07-6d577fd9706a\") " Oct 30 22:53:25.370355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.370227 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-credential-socket" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:25.370355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.370279 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-certs" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:25.370590 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.370358 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.370590 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.370388 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.370590 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.370355 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-socket" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:25.370590 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.370371 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istiod-ca-cert" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:53:25.371666 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.371639 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-envoy" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:53:25.372030 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.372008 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-kube-api-access-8zqfq" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "kube-api-access-8zqfq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:25.372123 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.372086 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-token" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:53:25.372357 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.372336 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-podinfo" pod "9e166eda-5b4a-4d02-9d07-6d577fd9706a" (UID: "9e166eda-5b4a-4d02-9d07-6d577fd9706a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:53:25.471225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.471193 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.471225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.471221 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.471225 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.471231 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zqfq\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-kube-api-access-8zqfq\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.471491 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.471241 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.471491 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.471249 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e166eda-5b4a-4d02-9d07-6d577fd9706a-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.471491 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.471258 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.471491 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.471274 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e166eda-5b4a-4d02-9d07-6d577fd9706a-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:53:25.923053 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.923017 5139 generic.go:356] "Generic (PLEG): container finished" podID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerID="4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad" exitCode=0 Oct 30 22:53:25.923242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.923079 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" Oct 30 22:53:25.923242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.923097 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" event={"ID":"9e166eda-5b4a-4d02-9d07-6d577fd9706a","Type":"ContainerDied","Data":"4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad"} Oct 30 22:53:25.923242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.923175 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x" event={"ID":"9e166eda-5b4a-4d02-9d07-6d577fd9706a","Type":"ContainerDied","Data":"e0084bdb3d9b6b6a505e9c2736ecf0e4a60083346f52231c8e252d8dc393d260"} Oct 30 22:53:25.923242 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.923199 5139 scope.go:122] "RemoveContainer" containerID="4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad" Oct 30 22:53:25.933195 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.933166 5139 scope.go:122] "RemoveContainer" containerID="5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e" Oct 30 22:53:25.941281 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.941263 5139 scope.go:122] "RemoveContainer" containerID="4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad" Oct 30 22:53:25.941548 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:53:25.941526 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad\": container with ID starting with 4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad not found: ID does not exist" containerID="4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad" Oct 30 22:53:25.941611 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.941555 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad"} err="failed to get container status \"4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad\": rpc error: code = NotFound desc = could not find container \"4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad\": container with ID starting with 4b8c79742b5cb4abaf401749afc8a3c4f9d6a28b17fb0eba481f9617bbb2a7ad not found: ID does not exist" Oct 30 22:53:25.941611 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.941573 5139 scope.go:122] "RemoveContainer" containerID="5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e" Oct 30 22:53:25.941823 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:53:25.941803 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e\": container with ID starting with 5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e not found: ID does not exist" containerID="5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e" Oct 30 22:53:25.941862 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:25.941831 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e"} err="failed to get container status \"5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e\": rpc error: code = NotFound desc = could not find container \"5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e\": container with ID starting with 5fcf5532b66fff74924852c6916321f0a9550b4422ffb4d275f1fc0983b2fb5e not found: ID does not exist" Oct 30 22:53:26.083248 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:26.083216 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x"] Oct 30 22:53:26.129739 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:26.129710 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xkc7x"] Oct 30 22:53:27.541811 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:53:27.541779 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" path="/var/lib/kubelet/pods/9e166eda-5b4a-4d02-9d07-6d577fd9706a/volumes" Oct 30 22:53:49.804318 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:53:49.804280 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:54:16.841347 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:16.841307 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt"] Oct 30 22:54:16.841936 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:16.841623 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" podUID="d63a8686-d9b5-40a7-8872-4ca63ddd1005" containerName="maas-api" containerID="cri-o://9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace" gracePeriod=30 Oct 30 22:54:17.096118 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.096060 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:54:17.162267 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.162232 5139 generic.go:356] "Generic (PLEG): container finished" podID="d63a8686-d9b5-40a7-8872-4ca63ddd1005" containerID="9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace" exitCode=0 Oct 30 22:54:17.162489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.162282 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" Oct 30 22:54:17.162489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.162319 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" event={"ID":"d63a8686-d9b5-40a7-8872-4ca63ddd1005","Type":"ContainerDied","Data":"9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace"} Oct 30 22:54:17.162489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.162381 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt" event={"ID":"d63a8686-d9b5-40a7-8872-4ca63ddd1005","Type":"ContainerDied","Data":"8d36d79fa068c86997a6a4525375ce4c1d0805e876ddc71382546370a5d4a758"} Oct 30 22:54:17.162489 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.162436 5139 scope.go:122] "RemoveContainer" containerID="9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace" Oct 30 22:54:17.166330 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.166309 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63a8686-d9b5-40a7-8872-4ca63ddd1005-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63a8686-d9b5-40a7-8872-4ca63ddd1005-openshift-service-ca\") pod \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " Oct 30 22:54:17.166460 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.166341 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63a8686-d9b5-40a7-8872-4ca63ddd1005-kube-api-access-p26dz\" (UniqueName: \"kubernetes.io/projected/d63a8686-d9b5-40a7-8872-4ca63ddd1005-kube-api-access-p26dz\") pod \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " Oct 30 22:54:17.166460 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.166369 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-metrics-certs\") pod \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " Oct 30 22:54:17.166593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.166474 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls\") pod \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\" (UID: \"d63a8686-d9b5-40a7-8872-4ca63ddd1005\") " Oct 30 22:54:17.166756 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.166731 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63a8686-d9b5-40a7-8872-4ca63ddd1005-openshift-service-ca" pod "d63a8686-d9b5-40a7-8872-4ca63ddd1005" (UID: "d63a8686-d9b5-40a7-8872-4ca63ddd1005"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:17.166856 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.166842 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d63a8686-d9b5-40a7-8872-4ca63ddd1005-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:17.168643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.168617 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-metrics-certs" pod "d63a8686-d9b5-40a7-8872-4ca63ddd1005" (UID: "d63a8686-d9b5-40a7-8872-4ca63ddd1005"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:17.168762 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.168652 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63a8686-d9b5-40a7-8872-4ca63ddd1005-kube-api-access-p26dz" pod "d63a8686-d9b5-40a7-8872-4ca63ddd1005" (UID: "d63a8686-d9b5-40a7-8872-4ca63ddd1005"). InnerVolumeSpecName "kube-api-access-p26dz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:17.168813 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.168798 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls" pod "d63a8686-d9b5-40a7-8872-4ca63ddd1005" (UID: "d63a8686-d9b5-40a7-8872-4ca63ddd1005"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:54:17.171954 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.171927 5139 scope.go:122] "RemoveContainer" containerID="9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace" Oct 30 22:54:17.172231 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:54:17.172208 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace\": container with ID starting with 9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace not found: ID does not exist" containerID="9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace" Oct 30 22:54:17.172297 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.172243 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace"} err="failed to get container status \"9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace\": rpc error: code = NotFound desc = could not find container \"9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace\": container with ID starting with 9aaf88b69459b5b53d8b400ae3454a91ba8a27240bc6fa21148c66fcef632ace not found: ID does not exist" Oct 30 22:54:17.267888 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.267853 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p26dz\" (UniqueName: \"kubernetes.io/projected/d63a8686-d9b5-40a7-8872-4ca63ddd1005-kube-api-access-p26dz\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:17.267888 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.267883 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:17.267888 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.267893 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d63a8686-d9b5-40a7-8872-4ca63ddd1005-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:17.586031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.585999 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt"] Oct 30 22:54:17.594318 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:17.594291 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-0c4786-5cfd6b9fbb-df8dt"] Oct 30 22:54:19.541518 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:19.541486 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63a8686-d9b5-40a7-8872-4ca63ddd1005" path="/var/lib/kubelet/pods/d63a8686-d9b5-40a7-8872-4ca63ddd1005/volumes" Oct 30 22:54:19.803793 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:54:19.803710 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:54:38.469782 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:38.469747 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9"] Oct 30 22:54:43.470312 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:43.470269 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" podUID="c162e80a-4ff7-49f6-8d7c-7de3b6f34877" containerName="payload-processing" containerID="cri-o://d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd" gracePeriod=25 Oct 30 22:54:44.014593 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:44.014552 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" podUID="c162e80a-4ff7-49f6-8d7c-7de3b6f34877" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:54:49.804224 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:54:49.804186 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:54:51.179624 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:51.179549 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl"] Oct 30 22:54:51.180164 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:51.179747 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" containerID="cri-o://073904ab591c812ca625ac41c950c8f0200459948bac439085a947e912cf7c3a" gracePeriod=30 Oct 30 22:54:54.014473 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:54.014405 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" podUID="c162e80a-4ff7-49f6-8d7c-7de3b6f34877" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:54:56.359573 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.359538 5139 generic.go:356] "Generic (PLEG): container finished" podID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerID="073904ab591c812ca625ac41c950c8f0200459948bac439085a947e912cf7c3a" exitCode=0 Oct 30 22:54:56.360031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.359580 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" event={"ID":"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3","Type":"ContainerDied","Data":"073904ab591c812ca625ac41c950c8f0200459948bac439085a947e912cf7c3a"} Oct 30 22:54:56.360031 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.359624 5139 scope.go:122] "RemoveContainer" containerID="6396bfd24a36d188da1c7106b61d04a242ffa2ad9daf0edfc90ec7ff6adc6d39" Oct 30 22:54:56.433833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.433803 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:54:56.561738 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.561700 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-credential-socket\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.561960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.561770 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istiod-ca-cert\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.561960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.561800 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-socket\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.561960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.561839 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-kube-api-access-527hx\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-kube-api-access-527hx\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.561960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.561875 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-data\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.561960 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.561914 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-envoy\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.562240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.561984 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-podinfo\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.562240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562028 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-credential-socket" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:56.562240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562058 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-token\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-token\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.562240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562090 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-certs\") pod \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\" (UID: \"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3\") " Oct 30 22:54:56.562240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562220 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-socket" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:56.562547 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562249 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istiod-ca-cert" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:54:56.562605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562569 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-certs" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:56.562605 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562587 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-data" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:56.562709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562497 5139 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:56.562709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562645 5139 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-credential-socket\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:56.562709 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.562660 5139 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istiod-ca-cert\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:56.564484 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.564442 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-token" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:56.564706 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.564687 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-kube-api-access-527hx" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "kube-api-access-527hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:54:56.564822 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.564802 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-podinfo" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 22:54:56.564939 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.564922 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-envoy" pod "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" (UID: "ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:54:56.663738 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.663705 5139 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-podinfo\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:56.663738 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.663733 5139 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-token\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:56.663738 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.663745 5139 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-workload-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:56.664024 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.663755 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-527hx\" (UniqueName: \"kubernetes.io/projected/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-kube-api-access-527hx\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:56.664024 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.663763 5139 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-data\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:56.664024 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:56.663772 5139 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3-istio-envoy\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:54:57.365275 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:57.365248 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" Oct 30 22:54:57.365740 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:57.365288 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl" event={"ID":"ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3","Type":"ContainerDied","Data":"2214e4e716a08d29ad291d9da122008ef584e1730eb5347e90cacd18858513ef"} Oct 30 22:54:57.365740 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:57.365342 5139 scope.go:122] "RemoveContainer" containerID="073904ab591c812ca625ac41c950c8f0200459948bac439085a947e912cf7c3a" Oct 30 22:54:57.543077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:57.543044 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl"] Oct 30 22:54:57.543077 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:57.543084 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-9af9e7-openshift-default-bc67b7949-wncsl"] Oct 30 22:54:59.541663 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:54:59.541631 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" path="/var/lib/kubelet/pods/ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3/volumes" Oct 30 22:55:03.589505 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:03.589480 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl"] Oct 30 22:55:03.735435 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:03.735354 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:55:03.832372 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:03.832330 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-kube-api-access-lnjdb\" (UniqueName: \"kubernetes.io/projected/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-kube-api-access-lnjdb\") pod \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\" (UID: \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\") " Oct 30 22:55:03.832576 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:03.832512 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-plugins-config-volume\") pod \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\" (UID: \"c162e80a-4ff7-49f6-8d7c-7de3b6f34877\") " Oct 30 22:55:03.832923 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:03.832898 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-plugins-config-volume" pod "c162e80a-4ff7-49f6-8d7c-7de3b6f34877" (UID: "c162e80a-4ff7-49f6-8d7c-7de3b6f34877"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:03.834561 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:03.834536 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-kube-api-access-lnjdb" pod "c162e80a-4ff7-49f6-8d7c-7de3b6f34877" (UID: "c162e80a-4ff7-49f6-8d7c-7de3b6f34877"). InnerVolumeSpecName "kube-api-access-lnjdb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:03.933450 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:03.933395 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:03.933450 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:03.933445 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnjdb\" (UniqueName: \"kubernetes.io/projected/c162e80a-4ff7-49f6-8d7c-7de3b6f34877-kube-api-access-lnjdb\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:04.398095 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.398062 5139 generic.go:356] "Generic (PLEG): container finished" podID="c162e80a-4ff7-49f6-8d7c-7de3b6f34877" containerID="d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd" exitCode=0 Oct 30 22:55:04.398259 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.398108 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" Oct 30 22:55:04.398259 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.398151 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" event={"ID":"c162e80a-4ff7-49f6-8d7c-7de3b6f34877","Type":"ContainerDied","Data":"d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd"} Oct 30 22:55:04.398259 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.398240 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9" event={"ID":"c162e80a-4ff7-49f6-8d7c-7de3b6f34877","Type":"ContainerDied","Data":"b58d2839dc5de14f60ac538e54874c479a4231552e1b67b6b6766aab14c4339c"} Oct 30 22:55:04.398406 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.398266 5139 scope.go:122] "RemoveContainer" containerID="d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd" Oct 30 22:55:04.407933 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.407918 5139 scope.go:122] "RemoveContainer" containerID="d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd" Oct 30 22:55:04.408172 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:55:04.408156 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd\": container with ID starting with d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd not found: ID does not exist" containerID="d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd" Oct 30 22:55:04.408221 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.408179 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd"} err="failed to get container status \"d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd\": rpc error: code = NotFound desc = could not find container \"d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd\": container with ID starting with d97d581cc86282d0e2311ed655168343b830ba72f433467962f5d8a09d949fcd not found: ID does not exist" Oct 30 22:55:04.565775 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.565739 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9"] Oct 30 22:55:04.568573 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:04.568542 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-9af9e7-7cf8fdb55f-wcff9"] Oct 30 22:55:05.541404 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:05.541367 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c162e80a-4ff7-49f6-8d7c-7de3b6f34877" path="/var/lib/kubelet/pods/c162e80a-4ff7-49f6-8d7c-7de3b6f34877/volumes" Oct 30 22:55:08.594931 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:08.594747 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" podUID="f93d2308-725f-402f-845e-041f586f08a6" containerName="payload-processing" containerID="cri-o://6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39" gracePeriod=25 Oct 30 22:55:08.892440 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:08.892323 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" podUID="f93d2308-725f-402f-845e-041f586f08a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:55:18.892563 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:18.892521 5139 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" podUID="f93d2308-725f-402f-845e-041f586f08a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 22:55:19.803926 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:55:19.803887 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:55:28.841177 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:28.841144 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:55:28.974953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:28.974866 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93d2308-725f-402f-845e-041f586f08a6-kube-api-access-8lfts\" (UniqueName: \"kubernetes.io/projected/f93d2308-725f-402f-845e-041f586f08a6-kube-api-access-8lfts\") pod \"f93d2308-725f-402f-845e-041f586f08a6\" (UID: \"f93d2308-725f-402f-845e-041f586f08a6\") " Oct 30 22:55:28.974953 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:28.974911 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93d2308-725f-402f-845e-041f586f08a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93d2308-725f-402f-845e-041f586f08a6-plugins-config-volume\") pod \"f93d2308-725f-402f-845e-041f586f08a6\" (UID: \"f93d2308-725f-402f-845e-041f586f08a6\") " Oct 30 22:55:28.975363 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:28.975336 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93d2308-725f-402f-845e-041f586f08a6-plugins-config-volume" pod "f93d2308-725f-402f-845e-041f586f08a6" (UID: "f93d2308-725f-402f-845e-041f586f08a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:28.977280 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:28.977251 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93d2308-725f-402f-845e-041f586f08a6-kube-api-access-8lfts" pod "f93d2308-725f-402f-845e-041f586f08a6" (UID: "f93d2308-725f-402f-845e-041f586f08a6"). InnerVolumeSpecName "kube-api-access-8lfts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:29.076769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.076721 5139 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93d2308-725f-402f-845e-041f586f08a6-plugins-config-volume\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:29.076769 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.076770 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lfts\" (UniqueName: \"kubernetes.io/projected/f93d2308-725f-402f-845e-041f586f08a6-kube-api-access-8lfts\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:29.508391 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.508348 5139 generic.go:356] "Generic (PLEG): container finished" podID="f93d2308-725f-402f-845e-041f586f08a6" containerID="6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39" exitCode=0 Oct 30 22:55:29.508629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.508450 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" Oct 30 22:55:29.508629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.508449 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" event={"ID":"f93d2308-725f-402f-845e-041f586f08a6","Type":"ContainerDied","Data":"6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39"} Oct 30 22:55:29.508629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.508592 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl" event={"ID":"f93d2308-725f-402f-845e-041f586f08a6","Type":"ContainerDied","Data":"b6545efef3c84878db10c03a9aca8f367550650eea265275510b4dd896b3f254"} Oct 30 22:55:29.508629 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.508615 5139 scope.go:122] "RemoveContainer" containerID="6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39" Oct 30 22:55:29.518443 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.518397 5139 scope.go:122] "RemoveContainer" containerID="6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39" Oct 30 22:55:29.518758 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:55:29.518735 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39\": container with ID starting with 6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39 not found: ID does not exist" containerID="6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39" Oct 30 22:55:29.518827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.518767 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39"} err="failed to get container status \"6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39\": rpc error: code = NotFound desc = could not find container \"6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39\": container with ID starting with 6ce44bc919a1de730c0b6f74bdc057fdff3d4516041dd909920d3e329dedbf39 not found: ID does not exist" Oct 30 22:55:29.674909 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.674868 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl"] Oct 30 22:55:29.681060 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:29.681027 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-6fc60c-5f97f5dc85-tmxkl"] Oct 30 22:55:31.541737 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:31.541703 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93d2308-725f-402f-845e-041f586f08a6" path="/var/lib/kubelet/pods/f93d2308-725f-402f-845e-041f586f08a6/volumes" Oct 30 22:55:40.465814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.465778 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms"] Oct 30 22:55:40.466252 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.465999 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" podUID="0124e363-c519-4839-8e70-6a01675c1fa6" containerName="maas-api" containerID="cri-o://afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75" gracePeriod=30 Oct 30 22:55:40.716955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.716891 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:55:40.888674 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.888638 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-metrics-certs\") pod \"0124e363-c519-4839-8e70-6a01675c1fa6\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " Oct 30 22:55:40.888883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.888705 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0124e363-c519-4839-8e70-6a01675c1fa6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0124e363-c519-4839-8e70-6a01675c1fa6-openshift-service-ca\") pod \"0124e363-c519-4839-8e70-6a01675c1fa6\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " Oct 30 22:55:40.888883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.888732 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls\") pod \"0124e363-c519-4839-8e70-6a01675c1fa6\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " Oct 30 22:55:40.888883 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.888774 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0124e363-c519-4839-8e70-6a01675c1fa6-kube-api-access-lvkt2\" (UniqueName: \"kubernetes.io/projected/0124e363-c519-4839-8e70-6a01675c1fa6-kube-api-access-lvkt2\") pod \"0124e363-c519-4839-8e70-6a01675c1fa6\" (UID: \"0124e363-c519-4839-8e70-6a01675c1fa6\") " Oct 30 22:55:40.889150 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.889125 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0124e363-c519-4839-8e70-6a01675c1fa6-openshift-service-ca" pod "0124e363-c519-4839-8e70-6a01675c1fa6" (UID: "0124e363-c519-4839-8e70-6a01675c1fa6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:55:40.890882 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.890856 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls" pod "0124e363-c519-4839-8e70-6a01675c1fa6" (UID: "0124e363-c519-4839-8e70-6a01675c1fa6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:40.891002 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.890969 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-metrics-certs" pod "0124e363-c519-4839-8e70-6a01675c1fa6" (UID: "0124e363-c519-4839-8e70-6a01675c1fa6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:55:40.891002 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.890987 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0124e363-c519-4839-8e70-6a01675c1fa6-kube-api-access-lvkt2" pod "0124e363-c519-4839-8e70-6a01675c1fa6" (UID: "0124e363-c519-4839-8e70-6a01675c1fa6"). InnerVolumeSpecName "kube-api-access-lvkt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:55:40.990370 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.990336 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:40.990580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.990374 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0124e363-c519-4839-8e70-6a01675c1fa6-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:40.990580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.990390 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0124e363-c519-4839-8e70-6a01675c1fa6-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:40.990580 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:40.990403 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvkt2\" (UniqueName: \"kubernetes.io/projected/0124e363-c519-4839-8e70-6a01675c1fa6-kube-api-access-lvkt2\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:55:41.564818 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.564783 5139 generic.go:356] "Generic (PLEG): container finished" podID="0124e363-c519-4839-8e70-6a01675c1fa6" containerID="afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75" exitCode=0 Oct 30 22:55:41.565238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.564832 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" Oct 30 22:55:41.565238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.564866 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" event={"ID":"0124e363-c519-4839-8e70-6a01675c1fa6","Type":"ContainerDied","Data":"afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75"} Oct 30 22:55:41.565238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.564931 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms" event={"ID":"0124e363-c519-4839-8e70-6a01675c1fa6","Type":"ContainerDied","Data":"c72e7659dbc4b3ed774016f5dfb020a073c300337a32bf7fbdd7e2681ed369f1"} Oct 30 22:55:41.565238 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.564966 5139 scope.go:122] "RemoveContainer" containerID="afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75" Oct 30 22:55:41.573350 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.573331 5139 scope.go:122] "RemoveContainer" containerID="afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75" Oct 30 22:55:41.573630 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:55:41.573612 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75\": container with ID starting with afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75 not found: ID does not exist" containerID="afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75" Oct 30 22:55:41.573729 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.573641 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75"} err="failed to get container status \"afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75\": rpc error: code = NotFound desc = could not find container \"afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75\": container with ID starting with afc72f34ef7fa50d0ff7f40198f1ab67d494a59de03f52c0bab1813ed8314f75 not found: ID does not exist" Oct 30 22:55:41.736909 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.736874 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms"] Oct 30 22:55:41.744530 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:41.744504 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-c29159-57d56d4dfb-xthms"] Oct 30 22:55:43.543030 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:43.542995 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0124e363-c519-4839-8e70-6a01675c1fa6" path="/var/lib/kubelet/pods/0124e363-c519-4839-8e70-6a01675c1fa6/volumes" Oct 30 22:55:49.803293 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:55:49.803258 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:55:58.796559 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:58.796522 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f"] Oct 30 22:55:58.847080 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:55:58.847052 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq"] Oct 30 22:56:13.817384 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:13.817332 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" containerID="cri-o://8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741" gracePeriod=45 Oct 30 22:56:13.865845 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:13.865805 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" containerID="cri-o://7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c" gracePeriod=45 Oct 30 22:56:14.110798 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.110772 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 22:56:14.126589 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.126567 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 22:56:14.209445 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209400 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-tmp-dir\") pod \"309f1902-d2a6-4c31-9ace-958449a06cde\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " Oct 30 22:56:14.209576 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209554 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-model-cache\") pod \"309f1902-d2a6-4c31-9ace-958449a06cde\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " Oct 30 22:56:14.209643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209612 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-home\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-home\") pod \"309f1902-d2a6-4c31-9ace-958449a06cde\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " Oct 30 22:56:14.209697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209634 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-tmp-dir" pod "309f1902-d2a6-4c31-9ace-958449a06cde" (UID: "309f1902-d2a6-4c31-9ace-958449a06cde"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:14.209697 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209668 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-dshm\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-dshm\") pod \"309f1902-d2a6-4c31-9ace-958449a06cde\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " Oct 30 22:56:14.209808 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209709 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/309f1902-d2a6-4c31-9ace-958449a06cde-tls-certs\" (UniqueName: \"kubernetes.io/secret/309f1902-d2a6-4c31-9ace-958449a06cde-tls-certs\") pod \"309f1902-d2a6-4c31-9ace-958449a06cde\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " Oct 30 22:56:14.209808 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209745 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/309f1902-d2a6-4c31-9ace-958449a06cde-kube-api-access-v2dx5\" (UniqueName: \"kubernetes.io/projected/309f1902-d2a6-4c31-9ace-958449a06cde-kube-api-access-v2dx5\") pod \"309f1902-d2a6-4c31-9ace-958449a06cde\" (UID: \"309f1902-d2a6-4c31-9ace-958449a06cde\") " Oct 30 22:56:14.209914 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209832 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-model-cache" pod "309f1902-d2a6-4c31-9ace-958449a06cde" (UID: "309f1902-d2a6-4c31-9ace-958449a06cde"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:14.209914 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.209845 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-home" pod "309f1902-d2a6-4c31-9ace-958449a06cde" (UID: "309f1902-d2a6-4c31-9ace-958449a06cde"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:14.210073 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.210052 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.210132 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.210074 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.210132 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.210088 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.211977 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.211954 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-dshm" pod "309f1902-d2a6-4c31-9ace-958449a06cde" (UID: "309f1902-d2a6-4c31-9ace-958449a06cde"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:14.212283 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.212236 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/309f1902-d2a6-4c31-9ace-958449a06cde-tls-certs" pod "309f1902-d2a6-4c31-9ace-958449a06cde" (UID: "309f1902-d2a6-4c31-9ace-958449a06cde"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:14.212367 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.212345 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/309f1902-d2a6-4c31-9ace-958449a06cde-kube-api-access-v2dx5" pod "309f1902-d2a6-4c31-9ace-958449a06cde" (UID: "309f1902-d2a6-4c31-9ace-958449a06cde"). InnerVolumeSpecName "kube-api-access-v2dx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:14.311261 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311234 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ede50429-a542-4b26-8921-7226330d50e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/ede50429-a542-4b26-8921-7226330d50e6-tls-certs\") pod \"ede50429-a542-4b26-8921-7226330d50e6\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " Oct 30 22:56:14.311473 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311297 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ede50429-a542-4b26-8921-7226330d50e6-kube-api-access-m84hh\" (UniqueName: \"kubernetes.io/projected/ede50429-a542-4b26-8921-7226330d50e6-kube-api-access-m84hh\") pod \"ede50429-a542-4b26-8921-7226330d50e6\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " Oct 30 22:56:14.311473 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311334 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-dshm\") pod \"ede50429-a542-4b26-8921-7226330d50e6\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " Oct 30 22:56:14.311473 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311358 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-home\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-home\") pod \"ede50429-a542-4b26-8921-7226330d50e6\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " Oct 30 22:56:14.311473 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311442 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-tmp-dir\") pod \"ede50429-a542-4b26-8921-7226330d50e6\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " Oct 30 22:56:14.311724 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311485 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-model-cache\") pod \"ede50429-a542-4b26-8921-7226330d50e6\" (UID: \"ede50429-a542-4b26-8921-7226330d50e6\") " Oct 30 22:56:14.311724 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311665 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-home" pod "ede50429-a542-4b26-8921-7226330d50e6" (UID: "ede50429-a542-4b26-8921-7226330d50e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:14.311807 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311711 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-tmp-dir" pod "ede50429-a542-4b26-8921-7226330d50e6" (UID: "ede50429-a542-4b26-8921-7226330d50e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:14.311856 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311836 5139 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-home\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.311897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311856 5139 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-tmp-dir\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.311897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311871 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/309f1902-d2a6-4c31-9ace-958449a06cde-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.311897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311870 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-model-cache" pod "ede50429-a542-4b26-8921-7226330d50e6" (UID: "ede50429-a542-4b26-8921-7226330d50e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:14.311897 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311885 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/309f1902-d2a6-4c31-9ace-958449a06cde-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.312047 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.311898 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2dx5\" (UniqueName: \"kubernetes.io/projected/309f1902-d2a6-4c31-9ace-958449a06cde-kube-api-access-v2dx5\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.313477 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.313455 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ede50429-a542-4b26-8921-7226330d50e6-kube-api-access-m84hh" pod "ede50429-a542-4b26-8921-7226330d50e6" (UID: "ede50429-a542-4b26-8921-7226330d50e6"). InnerVolumeSpecName "kube-api-access-m84hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:14.313828 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.313812 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ede50429-a542-4b26-8921-7226330d50e6-tls-certs" pod "ede50429-a542-4b26-8921-7226330d50e6" (UID: "ede50429-a542-4b26-8921-7226330d50e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:14.313886 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.313823 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-dshm" pod "ede50429-a542-4b26-8921-7226330d50e6" (UID: "ede50429-a542-4b26-8921-7226330d50e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 22:56:14.412852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.412776 5139 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ede50429-a542-4b26-8921-7226330d50e6-tls-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.412852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.412806 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m84hh\" (UniqueName: \"kubernetes.io/projected/ede50429-a542-4b26-8921-7226330d50e6-kube-api-access-m84hh\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.412852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.412816 5139 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-dshm\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.412852 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.412825 5139 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ede50429-a542-4b26-8921-7226330d50e6-model-cache\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:14.718662 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.718586 5139 generic.go:356] "Generic (PLEG): container finished" podID="309f1902-d2a6-4c31-9ace-958449a06cde" containerID="8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741" exitCode=0 Oct 30 22:56:14.718662 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.718625 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" Oct 30 22:56:14.718869 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.718671 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" event={"ID":"309f1902-d2a6-4c31-9ace-958449a06cde","Type":"ContainerDied","Data":"8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741"} Oct 30 22:56:14.718869 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.718713 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f" event={"ID":"309f1902-d2a6-4c31-9ace-958449a06cde","Type":"ContainerDied","Data":"2d55a4e21e34888e9d7426c069d37b43d734913823f4afdb40c1d470f63a8c22"} Oct 30 22:56:14.718869 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.718736 5139 scope.go:122] "RemoveContainer" containerID="8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741" Oct 30 22:56:14.720174 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.720147 5139 generic.go:356] "Generic (PLEG): container finished" podID="ede50429-a542-4b26-8921-7226330d50e6" containerID="7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c" exitCode=0 Oct 30 22:56:14.720276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.720180 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" event={"ID":"ede50429-a542-4b26-8921-7226330d50e6","Type":"ContainerDied","Data":"7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c"} Oct 30 22:56:14.720276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.720192 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" Oct 30 22:56:14.720276 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.720212 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq" event={"ID":"ede50429-a542-4b26-8921-7226330d50e6","Type":"ContainerDied","Data":"9c3333acd170b413b5cdea8b4aa9b770b70b8ab6b5398b03c84a4165473d0627"} Oct 30 22:56:14.728504 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.728489 5139 scope.go:122] "RemoveContainer" containerID="8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741" Oct 30 22:56:14.728763 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:56:14.728739 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741\": container with ID starting with 8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741 not found: ID does not exist" containerID="8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741" Oct 30 22:56:14.728841 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.728774 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741"} err="failed to get container status \"8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741\": rpc error: code = NotFound desc = could not find container \"8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741\": container with ID starting with 8a73b06898d9bfe5195db78e026569fdf7a70adbe3de90e790c4fbb662c12741 not found: ID does not exist" Oct 30 22:56:14.728841 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.728797 5139 scope.go:122] "RemoveContainer" containerID="7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c" Oct 30 22:56:14.736444 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.736410 5139 scope.go:122] "RemoveContainer" containerID="7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c" Oct 30 22:56:14.736724 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:56:14.736694 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c\": container with ID starting with 7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c not found: ID does not exist" containerID="7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c" Oct 30 22:56:14.736823 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.736728 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c"} err="failed to get container status \"7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c\": rpc error: code = NotFound desc = could not find container \"7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c\": container with ID starting with 7bbffdf5ef537807639293df805561cf48d9ea3834314117e391b7d4e151993c not found: ID does not exist" Oct 30 22:56:14.889281 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.889250 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq"] Oct 30 22:56:14.895258 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:14.895235 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-w0-9a5623-kserve-5478f97cff-rdptq"] Oct 30 22:56:15.006075 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:15.006045 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f"] Oct 30 22:56:15.012643 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:15.012616 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/distinct-2-w0-9a5623-kserve-54ccbff5cb-dpg5f"] Oct 30 22:56:15.541895 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:15.541865 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" path="/var/lib/kubelet/pods/309f1902-d2a6-4c31-9ace-958449a06cde/volumes" Oct 30 22:56:15.542266 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:15.542254 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ede50429-a542-4b26-8921-7226330d50e6" path="/var/lib/kubelet/pods/ede50429-a542-4b26-8921-7226330d50e6/volumes" Oct 30 22:56:19.803501 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:56:19.803461 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:56:20.243470 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.243411 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm"] Oct 30 22:56:20.243659 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.243636 5139 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" podUID="569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" containerName="maas-api" containerID="cri-o://995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42" gracePeriod=30 Oct 30 22:56:20.490562 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.490536 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 22:56:20.567013 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.566937 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls\") pod \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " Oct 30 22:56:20.567176 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.567033 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-kube-api-access-ktjvz\" (UniqueName: \"kubernetes.io/projected/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-kube-api-access-ktjvz\") pod \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " Oct 30 22:56:20.567176 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.567063 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-metrics-certs\") pod \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " Oct 30 22:56:20.567176 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.567108 5139 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-openshift-service-ca\") pod \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\" (UID: \"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e\") " Oct 30 22:56:20.567535 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.567505 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-openshift-service-ca" pod "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" (UID: "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 22:56:20.569139 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.569115 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-metrics-certs" pod "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" (UID: "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:20.569232 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.569204 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls" pod "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" (UID: "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 22:56:20.569271 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.569253 5139 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-kube-api-access-ktjvz" pod "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" (UID: "569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e"). InnerVolumeSpecName "kube-api-access-ktjvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 22:56:20.668138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.668108 5139 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-metrics-certs\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.668138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.668134 5139 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-openshift-service-ca\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.668138 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.668143 5139 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-maas-api-tls\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.668355 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.668152 5139 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktjvz\" (UniqueName: \"kubernetes.io/projected/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e-kube-api-access-ktjvz\") on node \"ip-10-0-133-5.us-west-2.compute.internal\" DevicePath \"\"" Oct 30 22:56:20.750231 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.750193 5139 generic.go:356] "Generic (PLEG): container finished" podID="569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" containerID="995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42" exitCode=0 Oct 30 22:56:20.750373 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.750246 5139 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" Oct 30 22:56:20.750373 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.750276 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" event={"ID":"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e","Type":"ContainerDied","Data":"995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42"} Oct 30 22:56:20.750373 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.750332 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm" event={"ID":"569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e","Type":"ContainerDied","Data":"5507b1b654caf4cdd55a60bcdfe63007fdccbf7f73c79f54f560eef5070bd119"} Oct 30 22:56:20.750373 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.750355 5139 scope.go:122] "RemoveContainer" containerID="995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42" Oct 30 22:56:20.759614 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.759591 5139 scope.go:122] "RemoveContainer" containerID="995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42" Oct 30 22:56:20.759908 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:56:20.759883 5139 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42\": container with ID starting with 995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42 not found: ID does not exist" containerID="995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42" Oct 30 22:56:20.760010 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.759913 5139 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42"} err="failed to get container status \"995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42\": rpc error: code = NotFound desc = could not find container \"995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42\": container with ID starting with 995f91b065b831a50363c03fbe48d3a413d23343fc2064ab8ffd796975229a42 not found: ID does not exist" Oct 30 22:56:20.916945 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.916835 5139 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm"] Oct 30 22:56:20.924343 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:20.924319 5139 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-9a5623-7df59d7fc5-xj6cm"] Oct 30 22:56:21.542143 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:21.542113 5139 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" path="/var/lib/kubelet/pods/569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e/volumes" Oct 30 22:56:33.783555 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:33.783523 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75b78c64d-gqk9g_7a96ed85-297f-4508-86bf-f4e6dca54950/manager/0.log" Oct 30 22:56:47.403827 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:47.403795 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75b78c64d-gqk9g_7a96ed85-297f-4508-86bf-f4e6dca54950/manager/0.log" Oct 30 22:56:49.802849 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:56:49.802810 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:56:50.406694 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:50.406664 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-56c8d9b444-ft4h4_1eeaef3a-f8b8-4502-a419-cad5c2b1d4ab/maas-api/0.log" Oct 30 22:56:57.728328 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:57.728300 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-794f59cf45-94n4w_e6e3b695-4e3c-4992-9485-f1c55585441a/kube-auth-proxy/0.log" Oct 30 22:56:58.650079 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:58.650043 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6jftz_79e8bb93-0439-4694-b7e8-8a0da249f331/payload-pre-processing/0.log" Oct 30 22:56:58.886085 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:58.886053 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ccz7d_9bbdcb9f-b9c8-46b3-8925-7c246f2affe3/payload-processing/0.log" Oct 30 22:56:59.980661 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:56:59.980627 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-6xqm9_7ab3c454-a14f-40db-ac93-87b411d35f90/main/0.log" Oct 30 22:57:00.253386 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:00.253353 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-6b74q_99d60883-524c-4d1b-a77a-2ffd15d3fbfd/main/0.log" Oct 30 22:57:00.519441 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:00.519346 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-phngz_a8587fc1-8061-4c87-913c-8f3f08a3108d/main/0.log" Oct 30 22:57:00.750705 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:00.750676 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-external-token-limit-simulated-kserve-65fc98cd9c-t7gcn_40bb43f0-83cd-4027-9870-1b48949a2ef2/main/0.log" Oct 30 22:57:00.985273 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:00.985187 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-zsvqf_d9284508-ee62-4a92-bcef-88e7cf77f265/main/0.log" Oct 30 22:57:01.217586 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:01.217557 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8d8gtr_d78c88db-4ef9-4d16-b3ed-9d8f4b291470/main/0.log" Oct 30 22:57:01.725844 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:01.725804 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-564c6_238c2b96-8487-422e-8fd3-9a757fe2169d/main/0.log" Oct 30 22:57:01.954185 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:01.954153 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-6w4fl_35879e2e-b0ac-4f01-ab06-79ace61c7f2b/main/0.log" Oct 30 22:57:12.631240 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631207 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-j9qjz/must-gather-qb95z"] Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631588 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631601 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631619 5139 cpu_manager.go:414] "containerMap: removing container" podUID="f60eb6af-5bbd-4a80-b835-60edc3466d68" containerName="maas-api" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631624 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60eb6af-5bbd-4a80-b835-60edc3466d68" containerName="maas-api" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631633 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631638 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631645 5139 cpu_manager.go:414] "containerMap: removing container" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631651 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631660 5139 cpu_manager.go:414] "containerMap: removing container" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631665 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631672 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631676 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631683 5139 cpu_manager.go:414] "containerMap: removing container" podUID="0124e363-c519-4839-8e70-6a01675c1fa6" containerName="maas-api" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631687 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0124e363-c519-4839-8e70-6a01675c1fa6" containerName="maas-api" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631696 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631700 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631707 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c162e80a-4ff7-49f6-8d7c-7de3b6f34877" containerName="payload-processing" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631711 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c162e80a-4ff7-49f6-8d7c-7de3b6f34877" containerName="payload-processing" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631716 5139 cpu_manager.go:414] "containerMap: removing container" podUID="569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" containerName="maas-api" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631720 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" containerName="maas-api" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631728 5139 cpu_manager.go:414] "containerMap: removing container" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631732 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631741 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631746 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631752 5139 cpu_manager.go:414] "containerMap: removing container" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631757 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631766 5139 cpu_manager.go:414] "containerMap: removing container" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" Oct 30 22:57:12.631757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631771 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631778 5139 cpu_manager.go:414] "containerMap: removing container" podUID="4a0e33b4-d4e4-468f-b9b4-c78654a41f13" containerName="payload-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631783 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a0e33b4-d4e4-468f-b9b4-c78654a41f13" containerName="payload-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631788 5139 cpu_manager.go:414] "containerMap: removing container" podUID="690d9602-55a9-41f5-8bf0-506dd9e9bcae" containerName="payload-pre-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631793 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690d9602-55a9-41f5-8bf0-506dd9e9bcae" containerName="payload-pre-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631800 5139 cpu_manager.go:414] "containerMap: removing container" podUID="f93d2308-725f-402f-845e-041f586f08a6" containerName="payload-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631804 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93d2308-725f-402f-845e-041f586f08a6" containerName="payload-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631812 5139 cpu_manager.go:414] "containerMap: removing container" podUID="d63a8686-d9b5-40a7-8872-4ca63ddd1005" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631816 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63a8686-d9b5-40a7-8872-4ca63ddd1005" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631826 5139 cpu_manager.go:414] "containerMap: removing container" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631830 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631835 5139 cpu_manager.go:414] "containerMap: removing container" podUID="4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631839 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631883 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631892 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ede50429-a542-4b26-8921-7226330d50e6" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631899 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a27a42-dfc1-491d-b955-774c21708323" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631904 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a0e33b4-d4e4-468f-b9b4-c78654a41f13" containerName="payload-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631908 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="690d9602-55a9-41f5-8bf0-506dd9e9bcae" containerName="payload-pre-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631913 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93d2308-725f-402f-845e-041f586f08a6" containerName="payload-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631918 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631923 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63a8686-d9b5-40a7-8872-4ca63ddd1005" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631930 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9249409a-b8c4-4b9b-908a-2402b39d8f4a" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631936 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c240dd4-fd65-4f9d-89d6-593d39ae0ebc" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631941 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631947 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60eb6af-5bbd-4a80-b835-60edc3466d68" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631951 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e166eda-5b4a-4d02-9d07-6d577fd9706a" containerName="istio-proxy" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631956 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba9092c-a737-48f9-99c4-675c8d0da106" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631961 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="a377639d-cd1a-4182-8067-fb2071641be1" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631967 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="0124e363-c519-4839-8e70-6a01675c1fa6" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631972 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c162e80a-4ff7-49f6-8d7c-7de3b6f34877" containerName="payload-processing" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631977 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="569d93b0-ea0b-4a4a-8d49-8e7e6af08a6e" containerName="maas-api" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.631982 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="309f1902-d2a6-4c31-9ace-958449a06cde" containerName="main" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.632049 5139 cpu_manager.go:414] "containerMap: removing container" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.632054 5139 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.632098 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1c4ca3-b3cb-4b0b-8456-4b3eca8aafa3" containerName="istio-proxy" Oct 30 22:57:12.632678 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.632105 5139 memory_manager.go:365] "RemoveStaleState removing state" podUID="c912fda5-cfef-4f0e-b9b4-8f6f6f44aeef" containerName="istio-proxy" Oct 30 22:57:12.635206 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.635188 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fwm4m/must-gather-qr8tb"] Oct 30 22:57:12.635351 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.635331 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j9qjz/must-gather-qb95z" Oct 30 22:57:12.638524 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.638506 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fwm4m/must-gather-qr8tb" Oct 30 22:57:12.693222 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.693199 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j9qjz\"/\"openshift-service-ca.crt\"" Oct 30 22:57:12.728932 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.728907 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6df304cd-adc0-4167-8c35-29e855b9aa86-must-gather-output\") pod \"must-gather-qb95z\" (UID: \"6df304cd-adc0-4167-8c35-29e855b9aa86\") " pod="openshift-must-gather-j9qjz/must-gather-qb95z" Oct 30 22:57:12.729062 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.728945 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5f7g\" (UniqueName: \"kubernetes.io/projected/46879b30-17b9-4b72-b684-dee904930d40-kube-api-access-c5f7g\") pod \"must-gather-qr8tb\" (UID: \"46879b30-17b9-4b72-b684-dee904930d40\") " pod="openshift-must-gather-fwm4m/must-gather-qr8tb" Oct 30 22:57:12.729122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.729062 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkdm\" (UniqueName: \"kubernetes.io/projected/6df304cd-adc0-4167-8c35-29e855b9aa86-kube-api-access-lfkdm\") pod \"must-gather-qb95z\" (UID: \"6df304cd-adc0-4167-8c35-29e855b9aa86\") " pod="openshift-must-gather-j9qjz/must-gather-qb95z" Oct 30 22:57:12.729122 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.729112 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/46879b30-17b9-4b72-b684-dee904930d40-must-gather-output\") pod \"must-gather-qr8tb\" (UID: \"46879b30-17b9-4b72-b684-dee904930d40\") " pod="openshift-must-gather-fwm4m/must-gather-qr8tb" Oct 30 22:57:12.743150 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.743128 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fwm4m\"/\"openshift-service-ca.crt\"" Oct 30 22:57:12.743269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.743169 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-j9qjz\"/\"default-dockercfg-vsf88\"" Oct 30 22:57:12.743269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.743128 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-j9qjz\"/\"kube-root-ca.crt\"" Oct 30 22:57:12.743269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.743128 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fwm4m\"/\"default-dockercfg-cxtk6\"" Oct 30 22:57:12.743269 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.743232 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fwm4m\"/\"kube-root-ca.crt\"" Oct 30 22:57:12.744658 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.744637 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j9qjz/must-gather-qb95z"] Oct 30 22:57:12.745684 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.745667 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fwm4m/must-gather-qr8tb"] Oct 30 22:57:12.830517 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.830484 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6df304cd-adc0-4167-8c35-29e855b9aa86-must-gather-output\") pod \"must-gather-qb95z\" (UID: \"6df304cd-adc0-4167-8c35-29e855b9aa86\") " pod="openshift-must-gather-j9qjz/must-gather-qb95z" Oct 30 22:57:12.830517 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.830518 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5f7g\" (UniqueName: \"kubernetes.io/projected/46879b30-17b9-4b72-b684-dee904930d40-kube-api-access-c5f7g\") pod \"must-gather-qr8tb\" (UID: \"46879b30-17b9-4b72-b684-dee904930d40\") " pod="openshift-must-gather-fwm4m/must-gather-qr8tb" Oct 30 22:57:12.830734 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.830713 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkdm\" (UniqueName: \"kubernetes.io/projected/6df304cd-adc0-4167-8c35-29e855b9aa86-kube-api-access-lfkdm\") pod \"must-gather-qb95z\" (UID: \"6df304cd-adc0-4167-8c35-29e855b9aa86\") " pod="openshift-must-gather-j9qjz/must-gather-qb95z" Oct 30 22:57:12.830792 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.830779 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/46879b30-17b9-4b72-b684-dee904930d40-must-gather-output\") pod \"must-gather-qr8tb\" (UID: \"46879b30-17b9-4b72-b684-dee904930d40\") " pod="openshift-must-gather-fwm4m/must-gather-qr8tb" Oct 30 22:57:12.830866 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.830846 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6df304cd-adc0-4167-8c35-29e855b9aa86-must-gather-output\") pod \"must-gather-qb95z\" (UID: \"6df304cd-adc0-4167-8c35-29e855b9aa86\") " pod="openshift-must-gather-j9qjz/must-gather-qb95z" Oct 30 22:57:12.831081 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.831067 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/46879b30-17b9-4b72-b684-dee904930d40-must-gather-output\") pod \"must-gather-qr8tb\" (UID: \"46879b30-17b9-4b72-b684-dee904930d40\") " pod="openshift-must-gather-fwm4m/must-gather-qr8tb" Oct 30 22:57:12.907878 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.907827 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkdm\" (UniqueName: \"kubernetes.io/projected/6df304cd-adc0-4167-8c35-29e855b9aa86-kube-api-access-lfkdm\") pod \"must-gather-qb95z\" (UID: \"6df304cd-adc0-4167-8c35-29e855b9aa86\") " pod="openshift-must-gather-j9qjz/must-gather-qb95z" Oct 30 22:57:12.908228 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.908206 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5f7g\" (UniqueName: \"kubernetes.io/projected/46879b30-17b9-4b72-b684-dee904930d40-kube-api-access-c5f7g\") pod \"must-gather-qr8tb\" (UID: \"46879b30-17b9-4b72-b684-dee904930d40\") " pod="openshift-must-gather-fwm4m/must-gather-qr8tb" Oct 30 22:57:12.950730 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.950699 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-j9qjz/must-gather-qb95z" Oct 30 22:57:12.955861 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:12.955841 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fwm4m/must-gather-qr8tb" Oct 30 22:57:13.347996 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:57:13.347963 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6df304cd_adc0_4167_8c35_29e855b9aa86.slice/crio-ae5d77cfa8b2449669f6abc548327567967ab3016c2427b7f5ec0f97c22d8ce4 WatchSource:0}: Error finding container ae5d77cfa8b2449669f6abc548327567967ab3016c2427b7f5ec0f97c22d8ce4: Status 404 returned error can't find the container with id ae5d77cfa8b2449669f6abc548327567967ab3016c2427b7f5ec0f97c22d8ce4 Oct 30 22:57:13.349703 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:13.349680 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-j9qjz/must-gather-qb95z"] Oct 30 22:57:13.350019 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:13.349997 5139 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 22:57:13.386200 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:57:13.386175 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46879b30_17b9_4b72_b684_dee904930d40.slice/crio-1fb6c1901ec31488b3d4a5c9a83340995650e10c4cc526fccd043b5eb7cd4f51 WatchSource:0}: Error finding container 1fb6c1901ec31488b3d4a5c9a83340995650e10c4cc526fccd043b5eb7cd4f51: Status 404 returned error can't find the container with id 1fb6c1901ec31488b3d4a5c9a83340995650e10c4cc526fccd043b5eb7cd4f51 Oct 30 22:57:13.388436 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:13.388398 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fwm4m/must-gather-qr8tb"] Oct 30 22:57:13.984133 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:13.984097 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fwm4m/must-gather-qr8tb" event={"ID":"46879b30-17b9-4b72-b684-dee904930d40","Type":"ContainerStarted","Data":"1fb6c1901ec31488b3d4a5c9a83340995650e10c4cc526fccd043b5eb7cd4f51"} Oct 30 22:57:13.985187 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:13.985163 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9qjz/must-gather-qb95z" event={"ID":"6df304cd-adc0-4167-8c35-29e855b9aa86","Type":"ContainerStarted","Data":"ae5d77cfa8b2449669f6abc548327567967ab3016c2427b7f5ec0f97c22d8ce4"} Oct 30 22:57:19.809768 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:57:19.809708 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:57:22.028154 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:22.028115 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fwm4m/must-gather-qr8tb" event={"ID":"46879b30-17b9-4b72-b684-dee904930d40","Type":"ContainerStarted","Data":"6de90d42c10f4eaea4bca6cc17fb29af613e13622b7c46932f3c99df1fd6b740"} Oct 30 22:57:22.028585 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:22.028170 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fwm4m/must-gather-qr8tb" event={"ID":"46879b30-17b9-4b72-b684-dee904930d40","Type":"ContainerStarted","Data":"34649d071065dbfc708925dfaf0c978a9a6582102b7db07499afcdeb9cc1aaab"} Oct 30 22:57:22.029834 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:22.029809 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9qjz/must-gather-qb95z" event={"ID":"6df304cd-adc0-4167-8c35-29e855b9aa86","Type":"ContainerStarted","Data":"ed9d350f9c4bb177cbfd025b21c5db740d3fa897df3014340113ef3f08566061"} Oct 30 22:57:22.029961 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:22.029844 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-j9qjz/must-gather-qb95z" event={"ID":"6df304cd-adc0-4167-8c35-29e855b9aa86","Type":"ContainerStarted","Data":"3532ce14012d9e8a18e00526dfd5d8711b03ed0872c5f1770b80ad6811e87f0d"} Oct 30 22:57:22.150520 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:22.149034 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fwm4m/must-gather-qr8tb" podStartSLOduration=1.917568647 podStartE2EDuration="10.149015683s" totalImagesPullingTime="8.231447036s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:22.147752177 +0000 UTC m=+4750.739037677" watchObservedRunningTime="2026-10-30 22:57:22.149015683 +0000 UTC m=+4750.740301171" Oct 30 22:57:22.264955 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:22.264897 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-j9qjz/must-gather-qb95z" podStartSLOduration=1.9258003019999999 podStartE2EDuration="10.264884559s" totalImagesPullingTime="8.339084257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:57:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:22.264036882 +0000 UTC m=+4750.855322383" watchObservedRunningTime="2026-10-30 22:57:22.264884559 +0000 UTC m=+4750.856170072" Oct 30 22:57:22.294983 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:22.294955 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-9rfmn_843e4020-8588-4ead-adda-ead6c9e0882b/global-pull-secret-syncer/0.log" Oct 30 22:57:25.217109 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:25.217070 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6knhd_788ef145-30de-43f5-8249-799dbb536b1b/konnectivity-agent/0.log" Oct 30 22:57:25.974524 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:25.974492 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75b78c64d-gqk9g_7a96ed85-297f-4508-86bf-f4e6dca54950/manager/0.log" Oct 30 22:57:26.170584 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:26.170552 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-133-5.us-west-2.compute.internal_3ab1a8fb7b394ed5c2ad3173d5294ab9/haproxy/0.log" Oct 30 22:57:26.176389 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:26.176368 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75b78c64d-gqk9g_7a96ed85-297f-4508-86bf-f4e6dca54950/manager/0.log" Oct 30 22:57:32.753115 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.753032 5139 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b"] Oct 30 22:57:32.763110 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.763075 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.830391 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.830364 5139 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v9zbs\"/\"default-dockercfg-r6fhw\"" Oct 30 22:57:32.849833 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.849802 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-sys\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.849970 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.849836 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-proc\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.849970 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.849906 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-podres\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.849970 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.849936 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wtk\" (UniqueName: \"kubernetes.io/projected/396b9d46-acc1-43ff-a14a-0b9a125a1d54-kube-api-access-q2wtk\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.850117 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.850041 5139 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-lib-modules\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.868847 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.868818 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b"] Oct 30 22:57:32.871187 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.871170 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v9zbs\"/\"openshift-service-ca.crt\"" Oct 30 22:57:32.881328 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.881304 5139 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v9zbs\"/\"kube-root-ca.crt\"" Oct 30 22:57:32.950645 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950617 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-podres\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.950835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950657 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wtk\" (UniqueName: \"kubernetes.io/projected/396b9d46-acc1-43ff-a14a-0b9a125a1d54-kube-api-access-q2wtk\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.950835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950697 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-lib-modules\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.950835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950732 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-sys\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.950835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950752 5139 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-proc\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.950835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950773 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-podres\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.950835 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950831 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-proc\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.951068 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950838 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-lib-modules\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:32.951068 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:32.950839 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/396b9d46-acc1-43ff-a14a-0b9a125a1d54-sys\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:33.012911 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:33.012836 5139 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wtk\" (UniqueName: \"kubernetes.io/projected/396b9d46-acc1-43ff-a14a-0b9a125a1d54-kube-api-access-q2wtk\") pod \"perf-node-gather-daemonset-dth2b\" (UID: \"396b9d46-acc1-43ff-a14a-0b9a125a1d54\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:33.075439 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:33.075397 5139 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:33.345562 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:33.345463 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75b78c64d-gqk9g_7a96ed85-297f-4508-86bf-f4e6dca54950/manager/0.log" Oct 30 22:57:33.442849 ip-10-0-133-5 kubenswrapper[5139]: W1030 22:57:33.442820 5139 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod396b9d46_acc1_43ff_a14a_0b9a125a1d54.slice/crio-403f0dfa13c9a0b31654623e71f92289c8183640f954c12b4a21cc1746477c3c WatchSource:0}: Error finding container 403f0dfa13c9a0b31654623e71f92289c8183640f954c12b4a21cc1746477c3c: Status 404 returned error can't find the container with id 403f0dfa13c9a0b31654623e71f92289c8183640f954c12b4a21cc1746477c3c Oct 30 22:57:33.444623 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:33.444601 5139 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b"] Oct 30 22:57:33.674691 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:33.674660 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75b78c64d-gqk9g_7a96ed85-297f-4508-86bf-f4e6dca54950/manager/0.log" Oct 30 22:57:34.086426 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:34.086377 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" event={"ID":"396b9d46-acc1-43ff-a14a-0b9a125a1d54","Type":"ContainerStarted","Data":"b916775ca623af96e3fb76f15ef2b0d3a4e0e6aa01af03291d8e4ab4a8920b53"} Oct 30 22:57:34.086893 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:34.086448 5139 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" event={"ID":"396b9d46-acc1-43ff-a14a-0b9a125a1d54","Type":"ContainerStarted","Data":"403f0dfa13c9a0b31654623e71f92289c8183640f954c12b4a21cc1746477c3c"} Oct 30 22:57:34.086893 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:34.086564 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:34.204167 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:34.204105 5139 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" podStartSLOduration=2.204088582 podStartE2EDuration="2.204088582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:34.203585706 +0000 UTC m=+4762.794871194" watchObservedRunningTime="2026-10-30 22:57:34.204088582 +0000 UTC m=+4762.795374069" Oct 30 22:57:40.100814 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:40.100784 5139 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-dth2b" Oct 30 22:57:40.358066 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:40.357984 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75b78c64d-gqk9g_7a96ed85-297f-4508-86bf-f4e6dca54950/manager/0.log" Oct 30 22:57:40.788996 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:40.788965 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-75b78c64d-gqk9g_7a96ed85-297f-4508-86bf-f4e6dca54950/manager/0.log" Oct 30 22:57:43.188540 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:43.188509 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28r28_a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11/node-exporter/0.log" Oct 30 22:57:43.597644 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:43.597615 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28r28_a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11/kube-rbac-proxy/0.log" Oct 30 22:57:44.006759 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:44.006732 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-28r28_a8e82ff8-c0cc-4e29-a5ba-27c4785b7a11/init-textfile/0.log" Oct 30 22:57:49.598802 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:49.598774 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-794f59cf45-94n4w_e6e3b695-4e3c-4992-9485-f1c55585441a/kube-auth-proxy/0.log" Oct 30 22:57:49.795803 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:49.795776 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-794f59cf45-94n4w_e6e3b695-4e3c-4992-9485-f1c55585441a/kube-auth-proxy/0.log" Oct 30 22:57:49.804173 ip-10-0-133-5 kubenswrapper[5139]: E1030 22:57:49.804145 5139 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fcsg4" containerName="extractor" Oct 30 22:57:50.446089 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:50.446052 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6jftz_79e8bb93-0439-4694-b7e8-8a0da249f331/payload-pre-processing/0.log" Oct 30 22:57:50.643566 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:50.643532 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-6jftz_79e8bb93-0439-4694-b7e8-8a0da249f331/payload-pre-processing/0.log" Oct 30 22:57:50.654069 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:50.654038 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ccz7d_9bbdcb9f-b9c8-46b3-8925-7c246f2affe3/payload-processing/0.log" Oct 30 22:57:50.852757 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:50.852726 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-ccz7d_9bbdcb9f-b9c8-46b3-8925-7c246f2affe3/payload-processing/0.log" Oct 30 22:57:51.152979 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:51.152902 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-z7xtk_fe3f0931-4f7d-4849-9c3d-f09e25aadc18/prometheus-operator/0.log" Oct 30 22:57:51.562817 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:51.562792 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-z7xtk_fe3f0931-4f7d-4849-9c3d-f09e25aadc18/kube-rbac-proxy/0.log" Oct 30 22:57:51.972846 ip-10-0-133-5 kubenswrapper[5139]: I1030 22:57:51.972759 5139 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-qrpk7_9a776136-2e97-427d-ad80-4b990b59d897/prometheus-operator-admission-webhook/0.log"